Deploy an LLM with Modal: From pip install to Production Endpoint

Deploy an LLM with Modal: From pip install to Production Endpoint

Most guides about self-hosting LLMs gloss over the hardest part: the infrastructure. You wrestle with CUDA drivers, manage Docker images, configure autoscaling, and somehow still end up paying for idle GPUs at 3 AM. Modal eliminates all of that. You write Python, you deploy, you get a URL.

This guide walks you through deploying an open-source LLM on Modal with vLLM as the inference engine. By the end, you'll have a live, OpenAI-compatible API endpoint running on H100 GPUs that scales to zero when nobody's using it.

What Is Modal (and Why Use It for LLMs)?

Modal is a serverless compute platform built specifically for AI workloads. Think AWS Lambda, but with GPU support, per-second billing, and a Python-native developer experience. There's no YAML, no Dockerfiles, no Kubernetes, you define your entire infrastructure in a Python script and deploy with a single command.

Here's why it's become the go-to for LLM deployment:

• Scale-to-zero billing, you pay nothing when your endpoint isn't handling requests
• Per-second GPU pricing, H100s at ~$3.95/hr, A100 80GB at ~$2.50/hr, billed by the second
• Sub-second cold starts, containers spin up fast, especially with memory snapshots
• $30/month free credits, enough to experiment without a credit card charge
• No DevOps, no Docker builds, no Terraform, no cluster management

If you've been running LLMs locally and want to give them a proper API without managing servers, Modal is the shortest path there.

Modal vs. RunPod vs. Lambda

Bottom line: Modal wins for bursty workloads and development. If your GPU utilization will consistently exceed 40%, a dedicated instance on RunPod or Lambda is cheaper. For everything else, prototyping, intermittent APIs, demos, Modal's scale-to-zero model saves real money.

Prerequisites

Before you start, you need three things:

1. Python 3.10+ installed locally
2. A Modal account, sign up free at modal.com
3. A Hugging Face account, for model access (most models are gated)

That's it. No GPU on your local machine, no CUDA toolkit, no Docker.

Step 1: Install Modal and Authenticate

Open a terminal and install the Modal Python package:

pip install modal

Then run the setup command to link your local environment to your Modal account:

modal setup

This opens a browser window for authentication. Once you confirm, Modal stores a token locally. You won't need to do this again.

Step 2: Define the Container Image

Modal containers are defined in Python. You specify the base image, install dependencies, and set environment variables, all as code. Create a file called app.py:

import modal

# Define the container image with CUDA, Python, and vLLM
vllm_image = (
    modal.Image.from_registry(
        "nvidia/cuda:12.8.0-devel-ubuntu22.04", add_python="3.12"
    )
    .entrypoint([])
    .pip_install(
        "vllm==0.13.0",
        "huggingface-hub==0.36.0",
    )
)

app = modal.App("llm-endpoint", image=vllm_image)

A few things to notice. There's no Dockerfile, that modal.Image chain replaces it entirely. The base image includes NVIDIA CUDA 12.8 with Ubuntu 22.04, and we install vLLM and the Hugging Face Hub client on top.

Step 3: Configure Model Storage with Volumes

LLM weights are large (a 7B parameter model is ~14 GB in fp16). You don't want to download them every time a container starts. Modal Volumes give you persistent storage that mounts directly into your containers:

# Persistent volumes for caching model weights
hf_cache = modal.Volume.from_name("huggingface-cache", create_if_missing=True)
vllm_cache = modal.Volume.from_name("vllm-cache", create_if_missing=True)

MODEL_NAME = "Qwen/Qwen3-4B-Thinking-2507-FP8"
MODEL_REVISION = "953532f942706930ec4bb870569932ef63038fdf"

We're using Qwen3-4B-Thinking (FP8) here, a quantized 4-billion parameter model that's fast, capable, and fits on a single GPU. You can swap this for any Hugging Face model: Llama 3.1 8B, Mistral 7B, or anything vLLM supports.

Why FP8? It cuts memory usage roughly in half compared to fp16, which means you can run bigger models on the same GPU, or run smaller models on cheaper GPUs. If you're curious about quantization trade-offs, our guide to running LLMs locally covers the precision formats in detail.

Step 4: Create the vLLM Server Function

This is where Modal's magic happens. You decorate a Python function with GPU requirements, scaling config, and a web server annotation. Modal handles everything else:

N_GPU = 1
MINUTES = 60
VLLM_PORT = 8000

@app.function(
    gpu=f"H100:{N_GPU}",
    scaledown_window=15 * MINUTES,
    timeout=10 * MINUTES,
    volumes={
        "/root/.cache/huggingface": hf_cache,
        "/root/.cache/vllm": vllm_cache,
    },
)
@modal.concurrent(max_inputs=32)
@modal.web_server(port=VLLM_PORT, startup_timeout=10 * MINUTES)
def serve():
    import subprocess

    cmd = [
        "vllm", "serve",
        MODEL_NAME,
        "--revision", MODEL_REVISION,
        "--served-model-name", MODEL_NAME,
        "--host", "0.0.0.0",
        "--port", str(VLLM_PORT),
        "--tensor-parallel-size", str(N_GPU),
        "--enforce-eager",  # Faster cold starts
    ]

    subprocess.Popen(" ".join(cmd), shell=True)

Let's break down the key decorators:

• gpu="H100:1", requests a single H100 GPU. Change to "A100-80GB:1" for cheaper inference, or "H100:2" for 70B+ models
• scaledown_window=15 * MINUTES, keeps the container warm for 15 minutes after the last request, then scales to zero
• @modal.concurrent(max_inputs=32), allows up to 32 concurrent requests per container (vLLM handles batching internally)
• @modal.web_server(port=8000), exposes the vLLM HTTP server directly as a Modal web endpoint
• --enforce-eager, skips CUDA graph compilation for faster cold starts (trade: slightly lower peak throughput)

The scaledown_window is your main cost lever. Set it to 5 minutes for dev, 15-30 minutes for production APIs where you expect regular traffic.

Step 5: Deploy to Production

One command. That's all:

modal deploy app.py

Modal builds the container image, pushes it to their registry, and returns a live URL:

✓ Created objects.
├── 🔨 Created mount /app.py
├── 🔨 Created volume huggingface-cache
├── 🔨 Created volume vllm-cache
└── 🔨 Created web function serve => https://your-workspace--llm-endpoint-serve.modal.run

The first deploy takes a few minutes because it downloads model weights into the volume. Subsequent deploys (and cold starts) are much faster since the weights are cached.

For development, use modal serve app.py instead, it hot-reloads on file changes and gives you a temporary URL.

Step 6: Call Your Endpoint (OpenAI-Compatible)

Your deployed vLLM server exposes an OpenAI-compatible API at /v1/chat/completions. You can use the standard OpenAI Python SDK to call it, just point the base URL at your Modal endpoint:

from openai import OpenAI

client = OpenAI(
    api_key="not-needed",  # vLLM doesn't require auth by default
    base_url="https://your-workspace--llm-endpoint-serve.modal.run/v1",
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-4B-Thinking-2507-FP8",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain what vLLM is in two sentences."},
    ],
    temperature=0.7,
    max_tokens=256,
)

print(response.choices[0].message.content)

This also works with curl:

curl -X POST https://your-workspace--llm-endpoint-serve.modal.run/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Thinking-2507-FP8",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 128
  }'

Any tool that supports an OpenAI-compatible API will work, LangChain, LlamaIndex, your own app. If you're routing requests across multiple LLM endpoints, an LLM gateway tool can help you manage failover and load balancing.

Cost Optimization Tips

Modal's per-second billing is already more efficient than hourly pricing, but you can squeeze more out of it:

1. Use FP8 Quantization

FP8 models use roughly half the VRAM of their fp16 counterparts. A Qwen3-8B in FP8 fits on a single H100, while the fp16 version needs most of that GPU's 80 GB. Less VRAM means you can use cheaper GPUs (A100 40GB, L40S) for smaller models.

2. Tune the Scaledown Window

The scaledown_window parameter controls how long a container stays warm after the last request:

3. Pick the Right GPU

Don't default to H100. Smaller models don't need it:

4. Enable Prompt Caching

If your workloads involve repeated system prompts or shared prefixes, vLLM's automatic prefix caching can significantly reduce latency and compute. You can enable it by adding --enable-prefix-caching to the vLLM serve command. For a deeper explore how caching works across different providers, check out our LLM prompt caching guide.

5. Use --enforce-eager for Cold Start Optimization

By default, vLLM compiles CUDA graphs on startup, which takes 1-3 extra minutes. The --enforce-eager flag skips this compilation. You trade ~10-15% peak throughput for dramatically faster cold starts. For bursty workloads where latency matters more than raw throughput, it's almost always the right call.

Going Beyond: Fine-Tuned Models

Once you're comfortable deploying base models, the natural next step is deploying your own fine-tuned version. The workflow is identical, you just point MODEL_NAME at your Hugging Face repo or a Modal volume containing your fine-tuned weights.

Modal also supports running fine-tuning jobs directly on their GPUs. You can train a LoRA adapter on Modal, save it to a volume, and deploy the merged model, all without leaving the platform. Our LLM fine-tuning guide covers the training side in depth.

The Complete app.py

Here's the full deployment script in one copy-paste-ready block:

import modal

# --- Image Definition ---
vllm_image = (
    modal.Image.from_registry(
        "nvidia/cuda:12.8.0-devel-ubuntu22.04", add_python="3.12"
    )
    .entrypoint([])
    .pip_install("vllm==0.13.0", "huggingface-hub==0.36.0")
)

# --- Volumes for Model Caching ---
hf_cache = modal.Volume.from_name("huggingface-cache", create_if_missing=True)
vllm_cache = modal.Volume.from_name("vllm-cache", create_if_missing=True)

# --- Model Config ---
MODEL_NAME = "Qwen/Qwen3-4B-Thinking-2507-FP8"
MODEL_REVISION = "953532f942706930ec4bb870569932ef63038fdf"

app = modal.App("llm-endpoint", image=vllm_image)

N_GPU = 1
MINUTES = 60
VLLM_PORT = 8000

@app.function(
    gpu=f"H100:{N_GPU}",
    scaledown_window=15 * MINUTES,
    timeout=10 * MINUTES,
    volumes={
        "/root/.cache/huggingface": hf_cache,
        "/root/.cache/vllm": vllm_cache,
    },
)
@modal.concurrent(max_inputs=32)
@modal.web_server(port=VLLM_PORT, startup_timeout=10 * MINUTES)
def serve():
    import subprocess

    cmd = [
        "vllm", "serve",
        MODEL_NAME,
        "--revision", MODEL_REVISION,
        "--served-model-name", MODEL_NAME,
        "--host", "0.0.0.0",
        "--port", str(VLLM_PORT),
        "--tensor-parallel-size", str(N_GPU),
        "--enforce-eager",
    ]

    subprocess.Popen(" ".join(cmd), shell=True)

Deploy with modal deploy app.py, swap MODEL_NAME for any Hugging Face model, and you're live.

Frequently Asked Questions

How much does it cost to run an LLM on Modal?

It depends on the GPU and how long your endpoint stays warm. A Qwen3-4B on an H100 costs ~$3.95/hr of active use. With scale-to-zero and a 15-minute scaledown window, a lightly-used endpoint might cost $5-15/month. The $30 free monthly credit covers a lot of experimentation.

Does Modal scale to zero?

Yes, that's one of its primary selling points. When no requests arrive for the duration of your scaledown_window, the container shuts down and you stop paying. The next request triggers a cold start (typically 2-10 seconds depending on model size and whether you use --enforce-eager).

Can I deploy Llama 3.1 or Mistral on Modal?

Absolutely. Swap the MODEL_NAME constant to any model vLLM supports: meta-llama/Llama-3.1-8B-Instruct, mistralai/Mistral-7B-Instruct-v0.3, or hundreds of others on Hugging Face. For 70B+ models, change N_GPU to 2 and use gpu="H100:2".

How do cold starts compare to RunPod?

Modal cold starts are typically 2-4 seconds for the container itself, plus model loading time. With model weights cached in a Volume and --enforce-eager enabled, you're looking at 10-30 seconds total for a 7-8B model. RunPod's serverless cold starts range from under 200ms (cached) to 6-12 seconds for larger containers, though their always-on model avoids cold starts entirely.

Is Modal's vLLM endpoint truly OpenAI-compatible?

Yes. vLLM implements the same /v1/chat/completions, /v1/completions, and /v1/models endpoints that OpenAI uses. You can point the official openai Python SDK at your Modal URL and it works out of the box. Streaming, function calling, and JSON mode all work.

Do I need a GPU on my local machine?

No. Your local machine just runs the Modal CLI. All GPU work happens on Modal's cloud infrastructure. You could deploy from a Chromebook if you wanted to.

How do I add authentication to my endpoint?

Modal web endpoints are public by default. For production, add a simple API key check in your application code, or use Modal's built-in web authentication features. You can also set up a proxy layer using an LLM gateway that handles auth, rate limiting, and routing.

What's the difference between modal serve and modal deploy?

modal serve creates a temporary endpoint that hot-reloads when you edit your code, perfect for development. modal deploy creates a persistent, production-ready endpoint with a stable URL. Use serve while iterating, deploy when you're ready to ship.

Can I use SGLang instead of vLLM?

Yes. Modal's documentation includes SGLang examples alongside vLLM. SGLang tends to have lower overhead for decode-heavy workloads and smaller models. vLLM is generally better for mixed workloads with heavy prefill. Both produce OpenAI-compatible endpoints.

How does this compare to deploying on Railway or Render?

Platforms like Railway, Render, and Fly.io are great for web apps, but they don't offer GPU instances. Modal is purpose-built for GPU workloads with per-second billing and autoscaling. If you need to serve an LLM, Modal (or RunPod) is the right tool, traditional PaaS platforms can't do it.

Sources

• Modal Documentation: High-Performance LLM Inference
• Modal: How to Deploy vLLM
• Modal Pricing
• vLLM OpenAI-Compatible Server Documentation
• Modal vLLM Inference Example (GitHub)