Fresh 2026

Gateway Routing Flow

How Merge Gateway routes a single LLM request across providers based on your routing policy.

Gateway gives you one endpoint for many model providers. A routing policy decides which provider and model serves each request, optimizing for cost, performance, or reliability without changing your application code.

flowchart TD
    A[App: POST /responses] --> B[Gateway receives request]
    B --> C{Routing policy}
    C -- Single provider --> D[Fixed model]
    C -- Priority --> E[Try primary, fail over to next]
    C -- Performance --> F[Pick fastest healthy provider]
    C -- Intelligent --> G[Score complexity 0-1]
    G --> H[Map score to model tier]
    D --> I[Provider call]
    E --> I
    F --> I
    H --> I
    I --> J{Healthy?}
    J -- No --> K[Failover per policy]
    K --> I
    J -- Yes --> L[Apply cost governance + logging]
    L --> M[Return completion]

Routing strategies

Strategy	Optimizes for	How it picks
Single provider	Simplicity	One fixed model
Priority	Reliability	Ordered list with failover
Performance	Latency	Fastest healthy provider
Intelligent	Cost vs quality	ML complexity score to model tier

See Routing Policies for each strategy, Cost governance and savings for budgets and spend caps, and Tool calling for function calls through Gateway.

Why a policy, not hardcoded models

The JSON you write defines the policy once. Application code keeps calling the same /responses endpoint. Swapping providers, adding failover, or shifting cost-vs-quality is a policy change, not a code change.