AI Model Routing Explained

AI model routing lets you send prompts to different large language models through a single API. An LLM router picks which model handles each request based on cost, speed, or capability. It sounds simple, but most teams do not realize where their data actually goes.

Companies adopt routing tools like OpenRouter, LiteLLM, and Portkey to cut costs and avoid vendor lock-in. The trade-off is control. When a router picks the cheapest model, it may silently send your prompt to a Chinese-hosted endpoint like DeepSeek, Qwen, or Moonshot AI.

This guide explains what AI model routing is, how routing decisions work, the full landscape of tools, and the compliance risk that regulated companies need to understand before adopting any router.

What Is AI Model Routing?

AI model routing is the practice of sending prompts to different LLMs through one unified interface. Instead of hard-coding a single model, your application sends every request to a routing layer. The router decides which model handles each prompt.

Think of it like a load balancer for language models. A web load balancer spreads traffic across servers. An LLM router spreads prompts across models from different providers.

The router can pick models based on cost, latency, context length, or capability. Some routers let you set rules. Others use algorithms to pick the cheapest model that clears a quality bar.

This approach grew popular in 2024 and 2025 as the number of available models exploded. Teams wanted to avoid lock-in to OpenAI or Anthropic and test new models without rewriting their code.

• One API endpoint replaces multiple provider integrations
• The router picks the model based on rules you set or defaults it chooses
• Fallback routing sends a prompt to a backup model if the first one fails
• Cost-based routing picks the cheapest model that meets a quality threshold
• Most routers normalize the API format so switching models needs no code change

Why Companies Use LLM Routers

Companies use LLM routers for three main reasons: cost savings, failover, and experimentation. Each reason is valid. The problem is that most teams stop there and never ask where their data actually lands.

Cost is the biggest driver. Different models charge different prices per token. A router that picks the cheapest model for simple tasks and reserves expensive models for hard tasks can cut token spend by 40 to 60 percent.

Failover is the second reason. If OpenAI goes down, the router sends prompts to Anthropic or Google instead. Your application stays up without manual intervention.

Experimentation is the third. Teams want to test new models, like DeepSeek R1 or Qwen 2.5, without changing production code. A router lets them A/B test models behind a stable API.

• Cost savings: route simple tasks to cheap models, hard tasks to expensive ones
• Failover: automatic switching when a provider has an outage
• Vendor flexibility: test new models without rewriting integration code
• Unified logging: one place to see all model usage across providers
• Rate-limit management: spread traffic across providers to avoid throttling

The Full Landscape of AI Routing Tools

The routing tool market splits into three categories: hosted marketplaces, open-source proxies, and cloud-provider gateways. Each category handles data residency and model selection differently.

Hosted marketplaces like OpenRouter aggregate models from dozens of providers into one API. You pay a markup on top of the underlying model price. OpenRouter lists over 200 models from more than 30 providers, including Chinese labs. For a full cost breakdown, see our OpenRouter pricing and LiteLLM pricing guides.

LiteLLM is an open-source proxy you self-host. It normalizes the API format across providers but leaves model selection and data routing to your configuration. Portkey is a hosted gateway with enterprise features like audit logs and guardrails.

Cloudflare AI Gateway and Vercel AI are infrastructure-layer gateways. They sit between your app and model providers, adding caching and logging. AWS Bedrock is Amazon's managed gateway with built-in data residency controls.

Smaller players include Unify, which benchmarks models and routes to the best one, and RouteLLM, a research project for cost-optimized routing.

• Hosted marketplaces: OpenRouter (200+ models, marketplace pricing)
• Open-source proxies: LiteLLM (self-hosted, format normalization)
• Enterprise gateways: Portkey (audit logs, guardrails, hosted)
• Infrastructure gateways: Cloudflare AI Gateway, Vercel AI (caching, logging)
• Cloud-provider gateways: AWS Bedrock, Databricks AI Gateway (data residency built in)
• Research tools: RouteLLM, Unify (benchmarking, quality-based routing)

How Routing Decisions Are Made

Routing decisions happen in one of four ways: manual selection, rule-based routing, cost optimization, or quality-based routing. The method your tool uses determines whether you have control over where data goes. For a broader look at how routing fits into the LLM stack, see our LLM orchestration explained guide.

Manual selection means you pick the model for each request. This is the safest approach for compliance because you always know which model runs. Most tools support this as a fallback.

Rule-based routing lets you set conditions. For example, route coding tasks to Claude and simple questions to a cheap model. The rules are explicit, so you can exclude specific providers or models.

Cost optimization is where the risk appears. The router picks the cheapest model that meets a capability threshold. If a Chinese-hosted model is the cheapest option, the router will use it unless you explicitly block it.

• Manual: you specify the model in each API call
• Rule-based: you define conditions that map task types to models
• Cost-optimized: the router picks the cheapest qualifying model automatically
• Quality-based: the router benchmarks models and picks the highest-scoring one for the task
• Fallback chains: the router tries models in order until one succeeds

The Compliance Blind Spot: Chinese-Hosted Models in Your Router

The compliance blind spot is this: most routing tools include Chinese-hosted models in their default catalog. Unless you explicitly exclude them, your data can be sent to servers in China without anyone on your team noticing.

DeepSeek is based in Hangzhou, China. Its privacy policy states that data may be stored on servers in the People's Republic of China. Under Chinese law, including the Personal Information Protection Law, National Security Law, and Intelligence Law, the government can compel access to data stored in China.

Other Chinese-hosted models in common router catalogs include Qwen (Alibaba), Yi (01.AI), and Moonshot AI (Kimi). When a router picks one of these models for cost reasons, it sends your prompt data to Chinese infrastructure.

For companies subject to ITAR, sending controlled technical data to Chinese servers is not just a policy violation. It can be a criminal offense under the Export Administration Regulations. HIPAA-covered entities face similar exposure if protected health information leaves approved infrastructure.

SOC 2 and ISO 27001 audits require that you document where data is processed. If your router is sending data to providers you have not vetted, your attestation may be inaccurate.

• DeepSeek, Qwen, Yi, and Moonshot AI are Chinese-hosted models found in most router catalogs
• China's PIPL, National Security Law, and Intelligence Law allow government access to data stored in China
• ITAR violations for sending controlled data to China carry criminal penalties
• HIPAA requires that PHI stays within approved infrastructure
• SOC 2 attestations require accurate documentation of data processing locations

Which Routing Tools Expose You to This Risk

Not all routing tools carry the same risk. The exposure depends on whether the tool includes Chinese-hosted models by default and whether it gives you controls to block them.

OpenRouter is the highest-risk category. It aggregates models from every provider, including DeepSeek, Qwen, and Moonshot AI. If you use cost-based routing or the auto-router, it can pick these models. Read the full breakdown in our OpenRouter explained guide.

LiteLLM is self-hosted, so the risk depends entirely on your configuration. If you add Chinese-hosted model endpoints to your proxy config, data goes there. If you do not, it does not. But the default documentation shows how to add DeepSeek, which normalizes the practice.

AWS Bedrock and Cloudflare AI Gateway are lower risk because they only expose models hosted in approved regions. Bedrock models run on AWS infrastructure, giving you region-level control. Cloudflare routes to provider endpoints but offers gateway-level controls.

Portkey falls in the middle. It is a hosted gateway with audit logs and the ability to restrict model access, but you must configure those restrictions yourself.

• Highest risk: OpenRouter (includes Chinese models by default, marketplace routing)
• Configuration-dependent: LiteLLM (self-hosted, you control the model list)
• Configuration-dependent: Portkey (hosted, but you must set model restrictions)
• Lower risk: AWS Bedrock (models run on AWS infrastructure, region controls)
• Lower risk: Cloudflare AI Gateway (gateway controls, no default Chinese models)

How to Audit Your AI Routing Stack

An audit starts with one question: which models can your routing tool send data to right now? If you cannot answer that in under five minutes, you have a gap.

Step one: list every model endpoint your router is configured to use. In OpenRouter, check your model preferences and any auto-routing settings. In LiteLLM, review your proxy config file. In Portkey, check your virtual keys and model access policies.

Step two: for each model, identify the hosting provider and the data processing location. If the model is hosted by a Chinese lab, assume data is stored in China unless the provider explicitly documents otherwise.

Step three: compare the data processing locations against your compliance requirements. ITAR, HIPAA, SOC 2, GDPR, and data residency laws each have specific rules about where data can go.

Step four: block or remove any models that violate your requirements. Then set up monitoring to catch configuration drift. A new team member adding DeepSeek for testing can undo your entire audit.

• List every model endpoint your router can reach
• Identify the data processing location for each model
• Map locations against ITAR, HIPAA, SOC 2, and data residency rules
• Block non-compliant models and document the decision
• Monitor for configuration drift so blocked models do not get re-added

How to Choose the Right Routing Tool

The right tool depends on your compliance posture, engineering capacity, and how much control you need over data flow. There is no single best answer.

If you are in a regulated industry, start with tools that offer data residency controls out of the box. AWS Bedrock and Cloudflare AI Gateway are the safest defaults because they do not include Chinese-hosted models by default.

If you need maximum flexibility and have a strong DevOps team, LiteLLM gives you full control because you self-host and configure every endpoint. But that control is only as good as your configuration discipline.

If you want the widest model selection and are not in a regulated industry, OpenRouter offers the most options. Just understand that the breadth includes Chinese-hosted models and plan accordingly.

• Regulated industries: start with AWS Bedrock or Cloudflare AI Gateway
• Strong DevOps teams: consider self-hosted LiteLLM with strict model allowlists
• Maximum flexibility: OpenRouter or Portkey with explicit model restrictions
• Enterprise: Portkey adds audit logs and guardrails on top of routing — see Portkey vs LiteLLM and Portkey alternatives
• Always: document which models are approved and block everything else