LiteLLM
Open SourceThe fastest, litest AI gateway.
Published 27 September 2026
Scores
Popularity4/5
56K+ GitHub stars and production adoption at Stripe, Google ADK, Greptile, and OpenHands make it the clear self-hosted leader in this genre.
Learning Curve2/5
Requires standing up and maintaining real infrastructure, PostgreSQL, Redis, the proxy server itself, meaningfully more setup than a zero-ops hosted aggregator like OpenRouter.
Flexibility5/5
Full control over routing logic, guardrails, virtual keys, and spend limits across 100+ providers, the most configurable option in the aggregator genre precisely because you own the deployment.
Performance4/5
8ms P95 latency at 1,000 RPS and a Rust core deliver genuinely fast proxy performance, though real-world throughput depends on the self-hosted infrastructure backing it.
Portability5/5
MIT licensed, genuinely self-hostable anywhere with Docker or the provided Terraform modules, no vendor lock-in to a hosted service.
About LiteLLM
LiteLLM is a self-hosted AI gateway — "the fastest, litest AI Gateway," a Rust core with a Python SDK — standing in for the same job OpenRouter does as a hosted service, but run entirely on your own infrastructure. It exposes a single, OpenAI-compatible interface across 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock, Gemini, and more), eliminating the need to juggle different SDKs, auth schemes, and request formats per vendor. With 56K+ GitHub stars, it's larger than every tool in the AI Runtime & Serving subcategory, effectively the anchor tool for self-hosted gateway infrastructure the way Ollama anchors self-hosted local inference.
Beyond routing, it ships genuine platform features: virtual API keys and per-project/per-user spend tracking, load balancing across providers, configurable guardrails, and an admin dashboard, plus deep observability integrations. Performance claims cite 8ms P95 proxy latency at 1,000 requests per second. Running it requires real infrastructure, PostgreSQL and Redis at minimum, with pre-built Terraform modules for AWS (ECS Fargate + Aurora + ElastiCache) and GCP (Cloud Run + Cloud SQL + Memorystore) covering the common deployment paths.
The core proxy is MIT licensed and free to self-host indefinitely, you pay only for your own infrastructure and whatever the underlying model providers charge. An optional Enterprise tier (SSO, Prometheus metrics, JWT auth, dedicated support, starting around $250/month and scaling toward $2,500/month for priority SLAs) sits on top for teams that want managed governance without giving up self-hosting. Adopted in production by Stripe, Google ADK, Greptile, and OpenHands.
Key Features
- Self-hosted, OpenAI-compatible gateway to 100+ LLM providers via one unified API
- Rust core with a Python SDK; 8ms P95 latency at 1,000 RPS
- Virtual API keys and spend tracking per project/user
- Load balancing, guardrails, and an admin dashboard for monitoring
- Deep observability integrations (LangFuse, OpenTelemetry, and others)
- Pre-built Terraform modules for AWS and GCP deployment
- MIT licensed core, free to self-host indefinitely
Pros
- MIT licensed and genuinely free to self-host, no feature gating on the core gateway functionality
- 56K+ GitHub stars, the clear self-hosted anchor for this genre, larger than every AI Runtime & Serving tool
- Full control over routing logic, spend limits, and data, all traffic stays on infrastructure you own
- Real production adoption at scale (Stripe, Google ADK, Greptile, OpenHands) validates it well beyond a hobby project
- Optional Enterprise tier adds governance/SSO without forcing a move off self-hosting
Cons
- Requires running and maintaining real infrastructure (PostgreSQL, Redis, the proxy itself), meaningfully more setup than a hosted aggregator like OpenRouter
- Enterprise pricing isn't published, requires contacting sales for SSO, dedicated support, and compliance-assistance features
- Self-hosting means you also own uptime and scaling, no built-in redundancy across cloud regions the way a managed aggregator provides by default
- No official model catalogue markup protection, provider price changes flow straight through since LiteLLM only proxies requests
LiteLLM Pricing
Open Source- · MIT licensed, free to self-host indefinitely, no license fee
- · Pay only for your own infrastructure and underlying model provider costs
- · Management UI, SSO integration, Prometheus metrics, JWT auth, LLM guardrails
- · Dedicated Slack support channel
- · Priority support with defined SLA response times, dedicated account management
- · Enhanced governance and compliance-certification assistance (SOC2, HIPAA)
Tech Stacks with LiteLLM
OpenCode Dev Stack
DeveloperOpenCode as the coding agent, in the terminal, the desktop app, an IDE extension, or the browser, running open-weight models such as DeepSeek, GLM, and Kimi, with your pick of GitHub or GitLab for version control.
Tools Related to LiteLLM
Works well with LiteLLM(1)
Groq is one of LiteLLM's 100+ documented supported providers — a self-hosted LiteLLM gateway can route requests to Groq's fast LPU-based inference alongside any other provider.
Alternatives to LiteLLM(1)
Both are OpenAI-compatible gateways to 100+ LLM providers. LiteLLM is self-hosted (MIT licensed, run it yourself), while OpenRouter is a zero-infrastructure hosted service — the same job, opposite deployment trade-off.
Learning Resources
No resources yet — check back soon.