LiteLLM Proxy: 1 API for 100+ LLMs (15-min Docker Setup)

LiteLLM Proxy: 1 API for 100+ LLMs (15-min Docker Setup)

Your team is sharing OpenAI API keys in Slack DMs. Nobody knows who spent $400 last Tuesday. There's no rate limiting, no fallback when a provider goes down, and switching from GPT-4o to Claude means changing code in twelve places. Sound familiar? A self-hosted LLM gateway fixes all of this, and LiteLLM proxy is the most popular open-source option, a single OpenAI-compatible endpoint that routes requests to 100+ LLM providers.

This guide covers the full litellm proxy setup: Docker Compose with PostgreSQL, virtual team keys with budgets, cost tracking, rate limits, and connecting AI IDEs like Claude Code and Cursor. If you're evaluating LLM gateway tools, this is the hands-on tutorial that goes from zero to production.

One important note before we start: LiteLLM's SDK (the Python library) and the Proxy Server are different things. The SDK is for a single developer calling multiple LLM APIs from Python. The proxy is for teams, it sits as a server between your apps and LLM providers. If you're a solo dev writing a script, the SDK is enough. If you're managing keys, budgets, and access for a team, you need the proxy. That's what we're setting up here.

LiteLLM Proxy at a Glance

Here's how the deployment methods compare:

For most teams, Docker Compose with PostgreSQL is the sweet spot. That's what we'll build toward, but first, let's get a proxy running in 60 seconds.

Prerequisites and Environment Setup

Before you start, make sure you have:

• Docker and Docker Compose installed (Docker Desktop includes both)
• At least one LLM API key (OpenAI, Anthropic, or a local Ollama instance)
• Basic terminal / CLI familiarity

Verify Docker is ready and export your API keys:

# Check Docker is installed
docker --version
docker compose version

# Export your LLM API keys (add to your shell profile for persistence)
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."

# Optional: set a master key for your proxy (you'll need this later)
export LITELLM_MASTER_KEY="sk-master-your-secret-key"

That's it. No special Python version, no OS-specific tooling. If Docker runs on your machine, you're good.

Quick Start, Your First LiteLLM Proxy in 60 Seconds

One command to start a proxy with GPT-4o:

docker run -d \
  --name litellm-proxy \
  -p 4000:4000 \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -e LITELLM_MASTER_KEY=$LITELLM_MASTER_KEY \
  ghcr.io/berriai/litellm:main-stable \
  --model openai/gpt-4o

Test it with curl:

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "Say hello from LiteLLM"}]
  }'

Or test from Python:

from openai import OpenAI

# Point the standard OpenAI SDK at your proxy
client = OpenAI(
    api_key="sk-master-your-secret-key",
    base_url="http://localhost:4000/v1"
)

response = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Say hello from LiteLLM"}]
)
print(response.choices[0].message.content)

What just happened? Your code talks to localhost:4000 using the standard OpenAI SDK format. The proxy receives the request, forwards it to OpenAI's API with the real key, and returns the response. Your application code never touches the actual API key.

That's the core idea. Now let's build a production setup.

Production Docker Compose Setup with PostgreSQL

The single docker run command works for testing, but production teams need persistent cost tracking, virtual keys, and proper database storage. That means Docker Compose with PostgreSQL.

The Docker Compose File

# docker-compose.yml
version: "3.9"

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    container_name: litellm-proxy
    ports:
      - "4000:4000"         # Proxy API port
    volumes:
      - ./config.yaml:/app/config.yaml   # Mount your config file
    environment:
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - DATABASE_URL=postgresql://litellm:litellm_password@postgres:5432/litellm
      - LITELLM_SALT_KEY=${LITELLM_SALT_KEY:-sk-salt-random-string}
    command: --config /app/config.yaml --detailed_debug
    depends_on:
      postgres:
        condition: service_healthy
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    container_name: litellm-db
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: litellm_password
    volumes:
      - litellm_pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 5s
      retries: 5
    restart: unless-stopped

volumes:
  litellm_pgdata:

The LITELLM_SALT_KEY encrypts virtual key data in the database. The LiteLLM production best practices docs recommend setting this for any team deployment.

Starting the Stack

# Create a .env file with your keys (don't commit this to git)
echo "LITELLM_MASTER_KEY=sk-master-your-secret" > .env
echo "OPENAI_API_KEY=sk-..." >> .env
echo "ANTHROPIC_API_KEY=sk-ant-..." >> .env
echo "LITELLM_SALT_KEY=sk-salt-$(openssl rand -hex 16)" >> .env

# Start everything
docker compose up -d

# Check logs
docker compose logs -f litellm

Verifying Everything Works

# Health check
curl http://localhost:4000/health

# Test a request
curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "ping"}]}'

If you see a successful response, your production stack is running. PostgreSQL stores all cost data, virtual keys, and usage metrics persistently across container restarts.

Verdict: Docker Compose + PostgreSQL is the recommended production setup. It gives you persistent storage, cost tracking, and virtual keys with about 10 minutes of work. The Docker deployment docs cover Kubernetes and Helm if you need auto-scaling later.

Config.yaml Walkthrough, A Real Multi-Provider Setup

Most tutorials show a config.yaml with one model. Here's what a real team config looks like with three providers, fallbacks, and load balancing.

The Config File

# config.yaml -- Real multi-provider setup
model_list:
  # Primary: OpenAI GPT-4o
  - model_name: gpt-4o          # The name YOUR code uses
    litellm_params:
      model: openai/gpt-4o      # The actual provider/model
      api_key: os.environ/OPENAI_API_KEY

  # Secondary: Anthropic Claude
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: os.environ/ANTHROPIC_API_KEY

  # Local: Ollama for development / cost-free testing
  - model_name: local-llama
    litellm_params:
      model: ollama/llama3.1
      api_base: http://host.docker.internal:11434

  # Fallback: route "gpt-4o" to Claude if OpenAI is down
  - model_name: gpt-4o
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  routing_strategy: least-busy    # Load balance across same-name models
  num_retries: 3
  retry_after: 5                  # Seconds between retries
  fallbacks: [{"gpt-4o": ["claude-sonnet"]}]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Model Aliases and Routing

Notice that gpt-4o appears twice in the config, once pointing to OpenAI, once to Anthropic. When your code requests gpt-4o, LiteLLM tries OpenAI first. If that fails, the fallbacks setting routes to Claude automatically. Your application code doesn't change at all.

If you're using production inference backends like vLLM or SGLang, you can add them the same way, just set the api_base to your inference server.

Provider Quick Reference

The routing_strategy: least-busy setting distributes requests across models with the same model_name. If you have two OpenAI keys (maybe different orgs with different rate limits), list them both under gpt-4o and LiteLLM balances the load.

Virtual Keys, Per-Team API Keys with Budgets and Rate Limits

This is where LiteLLM stops being "just a proxy" and becomes a team management tool. Virtual keys let you give each team member or service their own API key with spending limits and rate caps, all routed through your single set of provider API keys.

Creating a Team Key with Budget

# Create a virtual key with a $50/month budget
curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "team_id": "frontend-team",
    "max_budget": 50.0,
    "budget_duration": "1mo",
    "models": ["gpt-4o", "claude-sonnet"],
    "metadata": {"purpose": "frontend AI features"}
  }'

The response gives you a new key like sk-team-abc123.... Hand that to the frontend team. They can use it exactly like an OpenAI key, but it's limited to $50/month and only has access to the models you specified.

Setting Rate Limits

# Create a key with rate limits: 100 requests/minute, 50K tokens/minute
curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "team_id": "backend-team",
    "max_budget": 200.0,
    "budget_duration": "1mo",
    "rpm_limit": 100,
    "tpm_limit": 50000,
    "models": ["gpt-4o", "claude-sonnet", "local-llama"]
  }'

The virtual keys docs cover every parameter. You can also set per-user budgets and rate limits for even more granular control.

Monitoring Key Usage

import requests

# Check a key's current spend and limits
response = requests.get(
    "http://localhost:4000/key/info",
    headers={"Authorization": f"Bearer {MASTER_KEY}"},
    params={"key": "sk-team-abc123..."}
)
info = response.json()
print(f"Spent: ${info['spend']:.2f} / ${info['max_budget']:.2f}")
print(f"RPM used: {info['rpm_limit_used']} / {info['rpm_limit']}")

Need to revoke a compromised key? One API call:

curl -X POST http://localhost:4000/key/delete \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"keys": ["sk-team-abc123..."]}'

Verdict: Virtual keys are what make LiteLLM a team tool, not just a personal proxy. Without them, you're just adding a hop between your code and the LLM. With them, you have access control, budget enforcement, and usage attribution, the kind of stuff that keeps your CFO from panicking.

Cost Tracking and the LiteLLM Dashboard

Once PostgreSQL is connected, LiteLLM tracks every request's cost automatically. You don't need to configure anything, it knows the per-token pricing for every supported model.

The Dashboard

Access the built-in UI at http://localhost:4000/ui (log in with your master key). You'll see:

• Total spend across all teams and keys
• Per-model breakdown, which models are eating your budget
• Per-team spend, who's using what
• Request volume over time

For teams serious about reducing LLM API costs, the dashboard alone justifies running the proxy. You can also connect LiteLLM to external AI observability platforms like Langfuse or Helicone for deeper analytics.

Per-Provider Cost Comparison

Here's what the major models cost per million tokens (as of April 2026):

When you see these numbers in the dashboard broken down by team, the conversations about "should we use a cheaper model for this use case?" become very concrete.

Verdict: Cost tracking alone justifies the proxy for any team spending >$100/month on LLM APIs. You can't optimize what you can't measure.

Connecting AI IDEs, Claude Code, Cursor, and Continue

Here's something most LiteLLM guides skip entirely: you can point your AI coding tools at the proxy too. One proxy, all your IDE tools, unified billing.

Claude Code

# Set Claude Code to use your LiteLLM proxy
export ANTHROPIC_BASE_URL=http://localhost:4000/v1
export ANTHROPIC_API_KEY=sk-team-your-virtual-key

That's it. Claude Code sends requests to your proxy, which routes them to Anthropic (or wherever your config says) while tracking costs under your virtual key.

Cursor

In Cursor's settings, add a custom OpenAI-compatible endpoint:

{
  "openai.apiBaseUrl": "http://localhost:4000/v1",
  "openai.apiKey": "sk-team-your-virtual-key"
}

Continue (VS Code)

In Continue's config.json:

{
  "models": [
    {
      "title": "GPT-4o via LiteLLM",
      "provider": "openai",
      "model": "gpt-4o",
      "apiBase": "http://localhost:4000/v1",
      "apiKey": "sk-team-your-virtual-key"
    }
  ]
}

Why bother? Because now every developer's IDE usage goes through the proxy. You get per-person cost tracking for AI coding assistants, rate limits so nobody accidentally burns through $500 in a coding session, and a single place to switch models if you find a better option.

Troubleshooting Common Issues

"Config file not found"

This usually means the volume mount path is wrong in Docker. Make sure your config.yaml is in the directory you're mounting from:

# Check the file exists where you think it does
ls -la ./config.yaml

# The volume mount in docker-compose.yml should match
# volumes:
#   - ./config.yaml:/app/config.yaml

"Connection refused" to PostgreSQL

Docker networking catches everyone at least once. If LiteLLM can't reach Postgres, check that:

• The service name in DATABASE_URL matches the Docker Compose service name (postgres, not localhost)
• The depends_on with condition: service_healthy is set (so LiteLLM waits for Postgres to be ready)
• Both services are on the same Docker network (they are by default in Compose)

"Invalid API key format"

The most common confusion: your LITELLM_MASTER_KEY is for admin operations (creating virtual keys, accessing the dashboard). Virtual keys (sk-team-...) are what your applications use. Don't mix them up.

"Model not found"

The model field in your request must match a model_name in config.yaml. If your config defines gpt-4o but your code requests openai/gpt-4o, it won't match. Check the exact spelling.

Proxy starts but requests hang

Usually a firewall or port binding issue. Verify port 4000 is exposed and not blocked:

# Check if the port is listening
docker port litellm-proxy
# Should show: 4000/tcp -> 0.0.0.0:4000

Security: Avoid Versions 1.82.7 and 1.82.8

In March 2026, a supply chain incident affected LiteLLM versions 1.82.7 and 1.82.8. The compromised versions were pulled, and a clean release shipped at 1.83.0. Always pin your Docker image to a specific version and check the official security update before upgrading. If you're on 1.82.7 or 1.82.8, update immediately.

Which LiteLLM Setup Method Should You Choose?

If you're reading this guide for the first time, start with Docker Compose + PostgreSQL. You can always migrate to Kubernetes later, the config.yaml stays the same.

FAQ

What is LiteLLM proxy and how does it work?

LiteLLM proxy is an open-source AI gateway server that sits between your applications and LLM providers like OpenAI and Anthropic. It exposes a single OpenAI-compatible endpoint, so your code talks to one URL while the proxy handles routing, key management, cost tracking, and fallbacks behind the scenes.

How do I set up LiteLLM proxy with Docker Compose?

Create a docker-compose.yml with the LiteLLM proxy image and a PostgreSQL database, mount your config.yaml, set your API keys as environment variables, and run docker compose up -d. The Production Docker Compose section above has a complete, copy-paste-ready file.

How do I manage team API keys with LiteLLM?

Use virtual keys. Hit the /key/generate endpoint with your master key to create per-team or per-user keys. Each virtual key can have its own monthly budget, rate limits (RPM and TPM), and model access restrictions. The Virtual Keys section covers the full workflow.

How do I add cost tracking and rate limits to my LLM API?

Connect PostgreSQL to the proxy (via DATABASE_URL), and cost tracking happens automatically. For rate limits, set rpm_limit and tpm_limit when generating virtual keys. The built-in dashboard at /ui shows per-team and per-model spending.

Is LiteLLM proxy safe to use in production?

Yes, with one caveat: avoid versions 1.82.7 and 1.82.8, which were affected by a supply chain incident in March 2026. Use version 1.83.0 or later. Pin your Docker image version, set the LITELLM_SALT_KEY for encryption, and follow the official production best practices.

What is the difference between LiteLLM SDK and LiteLLM proxy?

The SDK is a Python library for calling multiple LLM APIs from your code. The proxy is a standalone server that your entire team connects to. Use the SDK when you're a solo dev writing a script. Use the proxy when you need shared access control, cost tracking, and rate limiting across a team.

Can I use LiteLLM proxy with Ollama and local models?

Absolutely. Add an entry to your config.yaml with model: ollama/llama3.1 and api_base: http://host.docker.internal:11434 (or your Ollama host). Your team can then access local models through the same proxy endpoint, which is great for development and cost-free testing.

How much does LiteLLM proxy cost?

LiteLLM proxy is free and open-source (MIT license). You self-host it on your own infrastructure. The only costs are your server (a small VPS is enough for most teams) and the LLM API costs you're already paying. BerriAI also offers a managed cloud version if you don't want to self-host.

What providers does LiteLLM support?

Over 100, including OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Google Vertex AI, Ollama, Hugging Face, Cohere, Replicate, and many more. The full list is on the LiteLLM GitHub repository.

How do I update LiteLLM proxy safely?

Always pin a specific version in your Docker image tag (e.g., ghcr.io/berriai/litellm:v1.83.2-stable). Before upgrading, check the changelog for breaking changes. Never use latest in production. And always verify the new version isn't on the security advisory list, the March 2026 incident proved that even trusted packages can be compromised.

Final Verdict and Next Steps

If your team spends money on LLM APIs and you don't have a proxy yet, start with Docker Compose + Postgres today. The setup takes 15 minutes, and you'll have cost visibility and access control by the end of it.

Once you're running, explore adding guardrails to your LLM pipeline for content filtering and safety checks. The proxy is the foundation, everything else builds on top of it.

Sources

• LiteLLM Proxy Quick Start, Official Docs
• LiteLLM Docker Deployment Guide
• LiteLLM Virtual Keys Documentation
• LiteLLM Production Best Practices
• LiteLLM Security Update, March 2026
• BerriAI/litellm, GitHub Repository