Fresh 2026

Multimodal

Supported multimodal content types (images and documents), model compatibility, and format translation across providers.

Gateway accepts multimodal content natively. Include image or document content blocks in your messages and Gateway routes to a capable model. No configuration needed. Gateway automatically detects which models support each modality and translates content to the provider's format.

Supported content types

Type	Content Blocks	Source Types	Example Models
Images	image, image_url	base64, URL	GPT-5.1, Claude Sonnet 4, Gemini 2.0 Flash
Documents	document	base64, URL	Claude Sonnet 4, Gemini 2.0 Flash

Quick example

from merge_gateway import MergeGateway

client = MergeGateway(api_key="YOUR_API_KEY")

response = client.responses.create(
    model="openai/gpt-5.1",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {"type": "image_url", "url": "https://example.com/photo.jpg"},
            ],
        }
    ],
)

print(response.output[0].content[0].text)

import { MergeGateway } from "merge-gateway-sdk";

const client = new MergeGateway({ apiKey: "YOUR_API_KEY" });

const response = await client.responses.create({
  model: "openai/gpt-5.1",
  input: [
    {
      type: "message",
      role: "user",
      content: [
        { type: "text", text: "What's in this image?" },
        { type: "image_url", url: "https://example.com/photo.jpg" },
      ],
    },
  ],
});

console.log(response.output[0].content[0].text);

Model compatibility

Gateway auto-detects multimodal capabilities from vendor-specific model metadata. Use GET /v1/models and inspect vendors..capabilities.input to see whether the route you plan to use supports image or document inputs.

Provider	Images	Documents
OpenAI	GPT-5.1, GPT-4o	None
Anthropic	Claude Sonnet 4, Claude Haiku 3.5	Claude Sonnet 4, Claude Haiku 3.5
Google	Gemini 2.0 Flash, Gemini 2.5 Pro	Gemini 2.0 Flash, Gemini 2.5 Pro
Bedrock	Varies by model	Varies by model

Context compression automatically protects multimodal messages. When trimming is needed, text-only messages are removed first, so your images and documents are preserved.

Next steps

Send images via URL or base64 to vision-enabled models

Send PDFs and documents to document-understanding models