Deploy an LLM on Serverless GPU: 5 Platforms, Real Prices, Honest Cold Starts

Deploy an LLM on Serverless GPU: 5 Platforms, Real Prices, Honest Cold Starts

RunPod charges $2.72/hr for an A100 80GB. Your endpoint gets twelve requests before lunch. That GPU sits idle the other 23 hours, billing the whole time. Deploy an LLM on serverless GPU and you pay only while a request runs. Five platforms do this. They bill in five different units. Nobody normalizes them.

An idle GPU costs exactly as much as a busy one.

Key Takeaways

• Serverless GPU bills only while a request runs and scales to zero between them.
• One GPU per instance on Cloud Run; 70B models need multi-GPU, so serverless usually cannot host them.
• Model weights live in the image, on a network volume, or get re-downloaded every cold start.
• Cold start is three things: container boot, weight load, engine init. Only the first is fast.
• Below roughly 47,000 requests a day, scale-to-zero beats a rented 24/7 GPU.

What does 'serverless GPU' actually mean for an LLM?

A serverless GPU platform runs your inference container on shared GPU hardware, spins it up when a request arrives, and scales to zero when traffic stops. You pay per second (or per minute, or per hour, depending on the vendor) only while the container is live. No idle bill. No reserved instance.

The billing unit differs by vendor, which is why the next section normalizes everything into $/GPU-hour.

Two constraints surprise people. First, Google Cloud Run allows one GPU per instance, maximum. That caps your VRAM ceiling at a single card. Second, "serverless" does not mean persistent state. There is no long-lived process holding your weights in RAM between requests. When the container dies, everything in memory dies with it. That single fact drives the storage decision in the model-weights section below.

Serverless doesn't mean no server. It means no server between your requests, and that's exactly where your model weights disappear to.

Which serverless GPU platform should you pick? (2026 prices, side by side)

We sell none of these platforms and take no affiliate revenue from any. Of the seven articles competing for this query and its close variants, four are published by a company that sells serverless GPU. This table isn't.

All rates read from the vendors' own pricing pages on 2026-07-30. Rates change; re-check before you commit.

The normalization arithmetic, shown once so you can audit it: Modal A100 80GB at $0.000694/s multiplied by 3600 seconds equals $2.4984/hr. Beam: $0.000625 times 3600 equals $2.25/hr. Baseten: $0.06667/min times 60 equals $4.00/hr. That 1.8x spread between Beam and Baseten for the same A100 is real, and it hides in plain sight because nobody publishes the same unit.

Three caveats. Beam's pricing page explicitly states it does not bill for server spin-up or container image load. Baseten's pricing FAQ answers "Do I pay for idle time on Baseten?" with "No, you do not pay for idle time," then adds that billable time is "the time your model is actively deploying, scaling up or down, or making predictions", so the meter covers more than prediction time, which is the part worth budgeting for. RunPod publishes per-hour rates and no cold-start figure on its pricing page.

If Modal is your pick, we've written the full Modal walkthrough separately.

Will your model even fit? VRAM, one-GPU limits and quota

Can you run a 70B model on a serverless GPU? Usually no. At FP16, 70B needs ~140 GB VRAM. Cloud Run caps at one GPU per instance (96 GB max on RTX PRO 6000). The math doesn't close without quantization (FP8/GGUF) or a multi-GPU platform.

Cloud Run's default quota is 3 L4 GPUs per region per project (RTX PRO 6000 is granted separately, as 3,000 milliGPU), across six L4 regions: asia-southeast1, asia-south1, europe-west1, europe-west4, us-central1, us-east4. That is the Cloud Run half of the data-residency answer for anyone asking about serverless GPU in Europe; Modal and RunPod document their own EU regions, and the FAQ has the rest.

Ismaili Simba's Cloud Run walkthrough on dev.to (April 2025) reports the quota request "can take some time (up to 5 working days) to be approved," and that "the L4 GPUs available on Cloud Run have a limit of 16GB RAM." Plan for that delay.

For model-by-model VRAM sizing, see our VRAM requirements guide.

Where do your model weights live (and what does that cost)?

A Reddit thread from November 2024 that still sat at #5 on Google for this exact query when we checked on 2026-07-30 asks, verbatim:

"I have been unable to find rate for storing the 80 gb model… What's an alternative if I don't want to download the model at every api calls (pod provisioned at call then closed)? … Why do these platforms not list model storage cost?"

Three low-score replies, none of them an answer. When we ran this search on 2026-07-30, the top ten results still included that two-year-old thread asking what model storage costs. Nobody in the thread answered it. Both platforms publish the rate. On Modal it's $0.09 per GiB per month with the first TiB free, so that 80 GB checkpoint bills $0.00. On RunPod it's $0.07 per GB per month for network storage under 1 TB, which puts the same checkpoint at roughly $5.60 a month.

The third row is the failure mode. A 2024 TU München review of ServerlessLLM (arXiv 2411.15664) reports LLaMA-2-70B (130 GB) needing 26+ seconds to pull at 5 GB/s, plus ~84 seconds to load onto 8 GPUs, against ~100 ms token generation. Those figures are cited in that review, not measured by it.

RunPod's pricing page carries a Storage section: container disk $0.10/GB/mo, volume disk $0.10/GB/mo running and $0.20/GB/mo idle, network storage $0.07/GB/mo under 1 TB and $0.05/GB/mo over it, high-performance network storage $0.14/GB/mo. The number exists. It just doesn't sit beside the per-GPU serverless rates a reader is comparing when the question occurs to them, nor in the endpoint configuration flow. A findability problem, not a secrecy one, and enough to keep the question alive two years later.

# Point the Hugging Face cache at a mounted network volume
# so weights persist across cold starts (RunPod / Modal pattern)
export HF_HOME=/workspace/hf-cache
export TRANSFORMERS_CACHE=/workspace/hf-cache
export HF_HUB_ENABLE_HF_TRANSFER=1

# Alternative: bake weights into the image at build time
FROM vllm/vllm-openai:latest
COPY ./model-weights /models/qwen3-27b-fp8
ENV MODEL_NAME=/models/qwen3-27b-fp8
# Downside: 30+ GB image, full rebuild to change models

If you haven't picked a model yet, our 2026 open-weights roundup sizes each option for deployment.

Deploy it: vLLM on RunPod Serverless, end to end

Six steps from zero to a callable endpoint. Verify each against RunPod's vLLM procedure (updated Jun 22 2026), which publishes no prices.

1. Pick your model. Choose an open-weights model sized to your GPU (see the VRAM table above). If it's gated on Hugging Face, generate an access token first.

2. Create the serverless endpoint. In RunPod's console, select Serverless, pick the vLLM worker template, and choose your GPU tier.

3. Set the environment variables that matter.

MODEL_NAME=Qwen/Qwen3.6-27B-FP8
MAX_MODEL_LEN=32768
GPU_MEMORY_UTILIZATION=0.90
DTYPE=auto

MODEL_NAME wrong means a 404 on boot. MAX_MODEL_LEN set too high for your VRAM means OOM before the first token. GPU_MEMORY_UTILIZATION above 0.95 leaves no headroom for KV-cache spikes.

1. Set min/max workers and idle timeout. Min workers at 0 gives you scale-to-zero (and cold starts). Min workers at 1 kills cold starts but bills continuously. The cold-start section below works through that trade-off.

2. Attach the network volume you decided on in the model-weights section, or accept the image-bake path. If you skip this, every cold start re-downloads the checkpoint.

3. Fire the first request. Read the endpoint ID from the console and confirm tokens come back.

curl -X POST "https://api.runpod.ai/v2/${ENDPOINT_ID}/runsync" \
  -H "Authorization: Bearer ${RUNPOD_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "input": {
      "messages": [{"role": "user", "content": "Explain cold starts in one sentence."}],
      "max_tokens": 60
    }
  }'

If you're weighing the serving engine itself, we compared vLLM and SGLang on throughput and latency.

How do you call the endpoint from your app?

Every platform in the shortlist speaks an OpenAI-compatible API. One Python snippet works everywhere. Switching provider is a base_url change, not a rewrite. That reframes "which provider" from a lock-in decision into a config decision.

import os

from openai import OpenAI

ENDPOINT_ID = os.environ["RUNPOD_ENDPOINT_ID"]

client = OpenAI(
    base_url=f"https://api.runpod.ai/v2/{ENDPOINT_ID}/openai/v1",
    api_key=os.environ["RUNPOD_API_KEY"],
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-27B-FP8",
    messages=[{"role": "user", "content": "What is scale to zero?"}],
    max_tokens=120,
)
print(response.choices[0].message.content)

# Same code, different provider. One line changes.
ENDPOINT_ID = os.environ["RUNPOD_ENDPOINT_ID"]

PROVIDERS = {
    "runpod": f"https://api.runpod.ai/v2/{ENDPOINT_ID}/openai/v1",
    "modal": "https://your-app--your-func.modal.run/v1",
    "beam": "https://your-beam-endpoint/v1",
}

client = OpenAI(base_url=PROVIDERS["modal"], api_key="your-key")

If every provider speaks OpenAI's dialect, "which serverless GPU provider" stops being an architecture decision and becomes one line of config.

Once you have multiple endpoints, our LLM gateway comparison covers routing and failover. For a lighter setup, a LiteLLM proxy adds retries and logging.

What cold start will you actually see?

For a 7B model on serverless GPU, expect 10 to 30 seconds on a true cold start (container boot plus weight load plus engine init), based on practitioner reports. Vendor docs quote 1 to 5 seconds for the container boot alone. The gap between those numbers is your checkpoint crossing a network.

RunPod's product page claims sub-200ms FlashBoot cold starts (a vendor claim, not a measurement). A practitioner in r/LLMDevs reports: "cold starts in my experience aren't great … I would say ~ 10 - 30 s", adding "most of my experience revolves around diffusion models though." Both are true. They measure different things.

The cold-start number is three numbers stacked:

1. Container boot. Modal's docs: "Containers boot in about one second." Cloud Run: instances with drivers pre-installed "start in approximately 5 seconds."

2. Weight load. The part nobody advertises. A 2024 TU München review of ServerlessLLM (arXiv 2411.15664) puts LLaMA-2-70B at 26+ seconds to pull plus ~84 seconds to load onto 8 GPUs.

3. Engine init. vLLM graph capture and warm-up. Logesh Umapathi measured Qwen3.6-27B-FP8 on an A100-80GB going from 460s baseline to 219s with eager mode to ~70s with vLLM sleep mode plus Modal GPU snapshots. That's 6.5x, published May 17 2026.

Our interpretation of published data: vendor figures time the container. Practitioner figures time the whole request. Between those two stopwatches sits 80 GB of weights crossing a network. The Type column is the point.

What to do: vLLM sleep mode, GPU snapshots, and tuning the scaledown window. Modal's default idle is 60 seconds (configurable 2s-20min). A warm worker kills cold starts but bills as always-on.

# Modal scaledown config (illustrative)
# A 60s window means you pay for 60 idle seconds per burst.
# A warm worker (min_containers=1) costs ~$2.50/hr on A100 80GB, 24/7.
@app.function(
    gpu="A100",
    scaledown_window=60,  # seconds idle before shutdown
    # min_containers=1,   # uncomment to kill cold starts; costs $60/day
)

Is serverless GPU cheaper than an always-on GPU?

Serverless GPU is cheaper when your GPU sits idle most of the day. At 3,000 requests/day averaging 2 seconds each on an A100 80GB, serverless costs roughly $4.17/day versus $65.28/day for a rented GPU running 24/7. The crossover is a duty-cycle question, not a volume question.

Assumptions (ours, stated so you can re-run them): A100 80GB at Modal's $2.50/hr, 2 seconds average GPU time per request, rented GPU at RunPod's $2.72/hr around the clock, hosted token API at $0.40/1M tokens (a mid-range published rate), and ~1,000 tokens per request.

The crossover lands around 47,000 requests/day. Below that, serverless wins. A hosted token API beats both at low volume with a commodity model. Break-even isn't about how many requests you get. It's about how many hours your GPU spends doing nothing.

BentoML's August 2024 analysis frames this trade-off well, though its pricing examples are GPT-3.5-turbo-era and two years stale.

For what a hosted token API costs at your volume, see our LLM API pricing comparison. To cut cost per request on the inference side, prompt caching typically saves 30-60% on repeated context.

# Break-even calculator: adjust these and re-run
REQUESTS_PER_DAY = 3000
SECONDS_PER_REQUEST = 2
SERVERLESS_RATE_HR = 2.50   # Modal A100 80GB
RENTED_RATE_HR = 2.72       # RunPod A100 80GB, 24/7

serverless_daily = REQUESTS_PER_DAY * SECONDS_PER_REQUEST / 3600 * SERVERLESS_RATE_HR
rented_daily = RENTED_RATE_HR * 24

print(f"Serverless: ${serverless_daily:.2f}/day")
print(f"Rented 24/7: ${rented_daily:.2f}/day")
print(f"Crossover: {int(RENTED_RATE_HR * 24 / (SECONDS_PER_REQUEST / 3600 * SERVERLESS_RATE_HR))} req/day")

When serverless GPU is the wrong call

• Sustained high traffic. Past ~47,000 requests/day at these rates, the duty cycle inverts and a rented GPU is cheaper per request.
• Hard sub-second SLOs on a cold path. No snapshot trick makes a first request instant. Keep a warm worker or rent the box.
• 70B+ models needing multiple GPUs. One GPU per instance on Cloud Run ends that conversation.
• Strict data residency. Six L4 regions is the whole menu on Cloud Run.
• Per-request economics that lose to a worker slot you're already paying for. If you have spare GPU headroom, adding inference costs nothing extra.

If your GPU is busy sixteen hours a day, serverless is the expensive option and anyone who tells you otherwise is selling serverless.

For the local-first path, see our local LLM tooling guide.

Frequently Asked Questions

What is a serverless GPU?

A serverless GPU platform runs your model container on shared hardware, scales to zero between requests, and bills only for active compute. You get an inference endpoint without managing a persistent GPU instance. The trade-off is a cold-start delay on spin-up.

How much does it cost to run an LLM on a serverless GPU?

An A100 80GB runs $2.25/hr on Beam, $2.50/hr on Modal, $2.72/hr on RunPod (verified 2026-07-30). Your bill depends on duty cycle: 3,000 requests/day at 2s each costs ~$4/day on Modal. Storage adds $0.09/GiB/month, with 1 TiB free.

Do I pay for storing the model weights?

Yes, but it's cheap. Modal charges $0.09/GiB/month with 1 TiB/month free, so an 80 GB checkpoint costs nothing. RunPod's pricing page lists network storage at $0.07/GB/month under 1 TB, about $5.60/month for the same 80 GB.

Does my model re-download on every request?

Only if nothing is cached. Weights on a persistent volume or baked into the image survive cold starts. Point the Hugging Face cache at ephemeral storage and a 130 GB model re-downloads every spin-up. That's the failure mode to avoid.

How long is the cold start for a 7B model?

Expect 10-30 seconds on a true cold start, per practitioner reports (r/LLMDevs, Dec 2024). The container boots in 1-5s (Modal, Cloud Run docs). The rest is weight loading and engine init. With snapshots and sleep mode, Umapathi measured ~70s for a 27B model, down from 460s.

Can I run a 70B model on serverless GPU?

Usually no. A 70B model at FP16 needs ~140 GB VRAM. Cloud Run allows one GPU per instance (96 GB max). You'd need FP8 quantization to fit on a single 80 GB card, or a multi-GPU platform. Most serverless surfaces cap at one GPU.

Is serverless GPU cheaper than renting a GPU 24/7?

Below ~47,000 requests/day (at 2s/request on an A100 80GB), yes. Serverless bills active seconds only; a rented GPU bills 24 hours regardless. Above that, the rented GPU wins. A hosted token API beats both at low volume. See the break-even table above.

Can I deploy an LLM on serverless GPU for free?

Modal gives $30/month free credits (Starter). Beam gives $30/month. GCP's $300 new-account credit covers Cloud Run GPU usage. Enough to prototype, not to run production traffic. No platform offers a permanent free tier for GPU inference.

Which serverless GPU providers are available in Europe?

Cloud Run serves L4 GPUs in europe-west1 (Belgium) and europe-west4 (Netherlands). Modal documents EU region selection (eu-west, eu-north, eu-south) priced at 1.5-1.75x base rates. RunPod names European data centers including EU-NL-1 and EU-FR-1. Pin the region explicitly; none of them defaults to the EU.

Do I need Docker to deploy an LLM on serverless GPU?

Not always. RunPod offers prebuilt vLLM worker templates that skip Docker. Modal builds containers from a Python image definition in code. For custom dependencies you'll write a Dockerfile. For stock vLLM serving, the prebuilt path works in minutes.

Conclusion

Five platforms, five billing units, one normalized table. The decision is simpler than the vendor pages make it look:

• Pick your GPU by VRAM, not by brand.
• Put your weights on a volume, not in the image, unless you never swap models.
• Expect 10-30 second cold starts and plan around them.
• Below ~47,000 requests/day, scale-to-zero wins on cost.
• The serving engine behind the endpoint is swappable; the base_url is one line.

You have a callable endpoint. Next: what sits in front of it when you need routing and failover? Our LLM gateway comparison answers that. Or talk through your setup with us.