A model that processes and reasons across more than one type of input — text, images, audio, video — within a single unified architecture rather than routing each modality to a separate specialist.
A multimodal model can accept and reason across multiple input types — text, images, audio, video, code — within a single forward pass. Rather than processing each modality in isolation and passing results between separate systems, a multimodal model builds a shared internal representation where information from different input types can interact. Gemini is natively multimodal. GPT-4o accepts text and images. Claude processes documents and images alongside text.
Each modality is first converted into tokens or embeddings that the model's shared architecture can process. Images are typically divided into patches, each encoded into an embedding vector; audio is converted into spectrograms or waveform representations. These encodings are then combined with text token embeddings and processed together through the transformer's attention layers. The model learns relationships between modalities — that the word "bridge" in a caption relates to the image of a bridge, for example — from training on paired examples.
Multimodal models affect how AI systems understand web content. Chrome's on-device models already process images for accessibility captions, phishing detection, and visual search classification. Gemini can receive images submitted alongside queries. As AI search becomes more visual, content that appears in images, charts, or video may receive different treatment from a multimodal grounding system than plain text would. The grounding snippet format DEJAN has documented is text-only today, but multimodal grounding — where image content feeds the answer — is an active development direction.
Multimodal models are also relevant to mechanistic interpretability research: Anthropic's sparse autoencoder work found that features like the "Golden Gate Bridge" concept activated for both textual mentions and images of the bridge, demonstrating that modalities share the same underlying representational space.
