Watch: Multimodal Model
A model that processes and reasons across more than one type of input — text, images, audio, video — within a single unified architecture rather than routing each modality to a separate specialist.
Transcript
Instead of processing text, images, and audio in separate silos, a multimodal model combines them into a single, shared system. It translates different types of media—like dividing an image into small patches or turning audio into waveforms—into a common language of mathematical vectors. This allows the model to connect the dots across different formats, understanding that a written caption about a bridge directly relates to the pixels in an accompanying photo.
This technology is already changing how we interact with the web. Google's Gemini, OpenAI's GPT-4o, and Anthropic's Claude can all process visual information alongside text. On-device models in browsers like Chrome are already using this capability for visual search, phishing detection, and generating accessibility captions.
As search engines become more visual, the way web content is indexed and retrieved is shifting. In the future, information locked inside an image, a chart, or a video clip may be just as searchable and influential as plain text.
We can even see this shared understanding in how these models think. Researchers have found that the exact same internal concept activates in a model's digital brain whether it reads the written name of a famous landmark or looks at a photograph of it. By unifying these different senses, multimodal models are creating a much richer, more cohesive way for artificial intelligence to understand our world.