LiteLLM Explained: Proxy, Security, and Enterprise Considerations

LiteLLM is an open-source LLM proxy that normalizes API calls across model providers. It lets you write one integration and route requests to OpenAI, Anthropic, Google, and dozens of other providers without changing your code.

It is also the LLM tool that had a serious security incident. In late 2024, a compromised version of the LiteLLM package on PyPI exfiltrated API keys from users who installed it. The issue was discovered and fixed, but it permanently changed the trust conversation around this tool.

This guide explains what LiteLLM is, how the proxy works, exactly what happened in the security incident, the current security status, and when enterprises should or should not use it.

What Is LiteLLM?

LiteLLM is an open-source Python library and proxy server built by BerriAI. It provides a unified API interface that translates your requests into the format each model provider expects.

You install it as a Python package or run it as a standalone proxy server. Your application sends OpenAI-formatted requests to the LiteLLM proxy. The proxy translates and forwards them to the provider you configured.

LiteLLM supports over 100 model providers. It handles API format translation, key management, load balancing, fallback routing, and spend tracking. The proxy mode adds a web UI for managing models and viewing usage.

Unlike OpenRouter, LiteLLM is self-hosted. You run it on your own infrastructure, which gives you full control over data flow. Your prompts do not pass through a third-party intermediary before reaching the model provider.

• Open-source Python library and proxy server
• Translates requests into each provider's API format
• Supports 100+ model providers including OpenAI, Anthropic, Google, and DeepSeek
• Self-hosted: you run it on your own infrastructure
• Key management, load balancing, fallback routing, and spend tracking built in

How the LiteLLM Proxy Works

The LiteLLM proxy sits between your application and model providers. It receives OpenAI-formatted API calls, translates them to the target provider's format, and forwards them. Responses come back through the same path.

You configure the proxy with a YAML file that lists your model providers, API keys, and routing rules. The proxy reads this config on startup and uses it to map incoming requests to the right provider.

The proxy supports virtual keys, which let you give each team or project its own API key with spend limits and model access restrictions. This is useful for cost control and access management across an organization.

You can run the proxy as a Docker container, a Python process, or behind a reverse proxy like Nginx. Most production deployments run it as a containerized service with a Postgres database for logging and key management.

• Receives OpenAI-formatted requests and translates to target provider format
• YAML config file defines providers, keys, and routing rules
• Virtual keys for per-team spend limits and model restrictions
• Runs as Docker container, Python process, or behind a reverse proxy
• Optional Postgres database for logging and key management

The LiteLLM Security Incident: What Happened

In late 2024, a malicious version of the LiteLLM package was published to PyPI, the Python package repository. This compromised version contained code that exfiltrated API keys from users who installed it. Anyone who installed the package during the affected window had their API keys stolen.

The attack was a supply chain compromise. The attacker gained the ability to publish a version of the package that looked legitimate but contained hidden malicious code. Users who ran pip install or updated their dependencies pulled the compromised version automatically.

The BerriAI team discovered and addressed the issue. The malicious version was removed from PyPI and a clean version was published. Users were advised to rotate all API keys that had been configured in LiteLLM during the affected period.

This incident is why people search "litellm compromised" and "litellm security." It was a real breach that affected real users. The package is clean now, but the incident raised lasting questions about supply chain trust for a tool that handles API keys for multiple providers.

• A malicious version was published to PyPI in late 2024
• The compromised version exfiltrated API keys from users
• Supply chain attack: users pulled the bad version through normal pip install
• BerriAI removed the malicious version and published a clean replacement
• All API keys configured during the affected period should have been rotated

Is LiteLLM Safe Now?

The compromised package was removed and replaced with a clean version. BerriAI continues to maintain and develop LiteLLM as an active open-source project. There have been no reported repeat incidents as of this writing.

However, the supply chain risk is inherent to any open-source package distributed through a public registry. The same attack vector exists for every pip-installable package, not just LiteLLM.

For enterprise users, the practical question is whether you trust the package distribution chain. Mitigations include pinning exact package versions, verifying checksums, using a private package mirror, and reviewing dependencies before updating.

The security of your LiteLLM deployment also depends on your configuration. If you add DeepSeek or other Chinese-hosted model endpoints to your proxy config, your data goes to those servers regardless of LiteLLM's own security posture.

• Compromised version removed, clean version published and maintained
• No repeat incidents reported as of mid-2026
• Supply chain risk is inherent to all public package registries
• Pin versions, verify checksums, and use private mirrors for production
• Configuration security is separate from package security

LiteLLM Compliance Considerations

Because LiteLLM is self-hosted, your compliance posture depends on how you deploy and configure it. The tool itself does not enforce any data residency rules. You do.

The proxy config file lists every model endpoint your instance can reach. If that config includes DeepSeek, Qwen, or any Chinese-hosted model, your data can be sent to servers in China. ITAR, HIPAA, and SOC 2 implications are the same as any other routing tool.

Virtual keys can restrict which models each team can use, which helps enforce policies. But there is no built-in compliance dashboard or automated audit trail. You need to build logging and monitoring around the proxy.

For HIPAA compliance specifically, you need a Business Associate Agreement (BAA) with every model provider your proxy sends data to. Self-hosting LiteLLM does not eliminate this requirement. It just means you manage the relationships directly instead of through a third-party gateway.

• Self-hosted means you own the compliance posture
• Proxy config defines which model endpoints your data can reach
• No built-in compliance dashboard or automated audit trail
• Virtual keys help restrict model access per team
• BAA required with each model provider for HIPAA workloads

When Enterprises Should Use LiteLLM

LiteLLM is a good fit when you have a DevOps team that can maintain self-hosted infrastructure and you need the flexibility to configure your own model endpoints and routing rules.

It works well for teams that want to standardize on one API format internally while using multiple providers behind the scenes. The format translation saves significant integration work across projects.

LiteLLM is also a good choice when you need to keep data off third-party gateways. Because it runs on your infrastructure, prompts go directly from your proxy to the model provider with no intermediary.

The spend tracking and virtual key features make it useful for organizations that need to allocate AI costs across teams or projects.

• You have DevOps capacity to run and maintain a self-hosted proxy
• You want one API format across multiple model providers
• You need data to flow directly to providers with no third-party intermediary
• You want per-team spend tracking and model access controls
• You are comfortable auditing and maintaining a YAML config file

When Enterprises Should Not Use LiteLLM

LiteLLM is a poor choice when you lack the engineering resources to maintain a self-hosted proxy, need turn-key compliance features, or cannot commit to ongoing security monitoring of your deployment.

If the 2024 supply chain incident is a dealbreaker for your security team, that is a valid position. The risk was real, and the trust is hard to rebuild. Consider alternatives with a different distribution model.

If you need built-in audit logging, compliance dashboards, and role-based access controls without building them yourself, look at Portkey or AWS Bedrock instead.

If your team is small and you just need multi-model access without managing infrastructure, OpenRouter or Cloudflare AI Gateway may be simpler choices, with the data residency caveats noted elsewhere in this cluster.

• No DevOps capacity for self-hosted infrastructure
• Need turn-key compliance features and audit dashboards
• Supply chain trust is a dealbreaker for your security team
• Small team that cannot maintain config and security monitoring
• Prefer managed services over self-hosted tools