Current artificial intelligence (AI) systems are unimodal: they process information from one modality, such as text or images.
The next step in AI is multimodal AI systems, which can receive and process inputs from and to multiple modalities such as sounds, images, text, and video.
Multimodal AI systems will revolutionize search in the short term and bring AI into the physical world.
What Is Multimodal AI?
As humans, we are able to easily distinguish between various forms of media such as text, images, or video which have different meanings. Current AI systems can’t do this.
However, the next evolution in AI systems, multimodal AI systems, can simultaneously process different data types (such as text, images, video, speech, and numerical data) to provide better classifications, predictions, recommendations, and information.
To best solve a problem or present accurate information, multimodal AI systems associate the same concept or object over different types of scenarios and media.
For example, a multimodal AI system will pick up on a specific concept – such as a basketball – in different contexts. Whether shown in a picture, in a video, described in writing, or referred to abstractly, the system can understand and express the concept in various forms and integrate it with other concepts.
When presented with real-world problems, Multimodal AI can outperform unimodal AI. Multimodal AI systems have better contextual understanding, improved accuracy, and can therefore offer more seamless, natural interactions.
How Multimodal AI Works
Multimodal AI architecture consists of three components:
- Unimodal encoders for each input modality
- A fusion network for combining the features of the different modalities
- A classifier for making predictions based on the fused data
Multiple unimodal encoders put together create a multimodal network. In a process known as ‘encoding’, each unimodal encoder processes its respective inputs separately. For example, one encoder could be processing textual data while another could be processing visual data.
After the unimodal encoding is complete, the refined insights and data are extracted from each model and then combined. Multiple fusion processes have been proposed and implemented. The multimodal data fusion step is essential for the effectiveness of the model.
Lastly, the ‘decision’ network receives and accepts the fused and encoded data and gets trained on how to best perform the specific task.
Multimodal AI Technology Stack
Multimodal AI systems will require the following technology stack:
- Natural language processing technologies for speech recognition, so that
the system can make sense of and transcribe spoken language, opening up the system to voice commands.
the system can make sense of and transcribe spoken language, opening up the system to voice commands.
- Computer vision technologies for image and video recognition, so that the system can analyze and interpret complex visual data and contextualize activities, objects, and people.
- Textual analysis so that the system can understand written text including language translation and sentiment analysis.
- Speed processing and data mining technologies to be able to compute results quickly in real-time.
- Multimodal integration, so that the system can combine multiple inputs across modalities and form a more complete understanding of a given situation.
Industry Applications of Multimodal AI
Search is the first major application of multimodal AI.
One version of multimodal search is an expansion of services like the ChatGPT-powered Bing that have mushroomed across the internet. Search engines that can turn text into images, describe why an image is funny, or generate a video from an image, are all likely to be early and fast-improving examples of multimodal AI.
Another version is corporate applications of search. For example, if your company has referred to insights from a thought leader called Emily in various Google Docs and Spreadsheets, and the business leader’s insights are also available in public forums like Youtube and articles, a multimodal AI system can scan all of these, make conceptual connections between them, and present them in different formats (like text or video outputs).
Emily's Data Across the Internet, which Multimodal AI systems will be able to understand and contextualize.
Beyond search, there are many other use cases that multimodal AI solutions could be ideal for:
- Automated virtual assistants
- Automated customer service
- Automotive sector solutions including human-machine interfaces, driver assist systems, and autonomous driving solutions
- Drones
- Healthcare diagnosis solutions
- Media and entertainment solutions
- Personalized advertising and marketing systems
- Predictive maintenance of complex industrial systems
- Product design
- Robotic process automation
- Security and surveillance
- Smart home solutions
Conclusion
Multimodal AI systems will process data, understand the world, and express itself more closely to how we do. In the short term, it will revolutionize search. In the long term, it will bring AI systems out of our computers, phones, and smart speakers, and into the physical world around us.
Also published here.