Run Embedding Models Locally with Ollama: I Timed Cold vs Warm GPU

Run Embedding Models Locally with Ollama: I Timed Cold vs Warm GPU

You can run embedding models locally with Ollama and stop paying OpenAI $0.02 per million tokens for every chunk you index. The trade: you own the GPU, the cold starts, and the ops. Ollama serves them on port 11434 with no API key. Here's the full workflow, from ollama pull to a warm vector search that answers queries.

Key Takeaways

• Ollama serves embeddings locally on http://localhost:11434 via POST /api/embed, with no API key and $0 per token.
• Use /api/embed (current, batch array); /api/embeddings is legacy and the usual 404 source.
• Popular local models: nomic-embed-text (768-dim), mxbai-embed-large (1024), bge-m3 (1024), embeddinggemma (768).
• Match your embedding dimension to your vector DB column, and pin the model with keep_alive to skip cold-start latency.

What Do You Need to Run Embeddings Locally with Ollama?

Everything you need to run embeddings locally is three pieces: an embedding model, the Ollama server on port 11434, and a vector store to hold the output. Ollama downloads and serves the model; your code posts text to /api/embed; the vectors land in a database like pgvector, Qdrant, or Chroma. No cloud round-trip, no per-token bill.

Two commands get you a working embedding in under a minute:

ollama pull nomic-embed-text
curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": "The quick brown fox"
}'

That's the whole quickstart. The rest of this tutorial fills in the model choice, the store, and the two gotchas that bite everyone: the endpoint confusion and the cold-start penalty.

Step 1: Install Ollama and Pull an Embedding Model

Install Ollama, confirm the server is listening on port 11434, then pull an embedding model. Ollama runs as a background service, so ollama pull nomic-embed-text downloads the weights and the next /api/embed call serves them. Embedding models are tiny next to chat models, so this is fast.

# macOS / Linux install
curl -fsSL https://ollama.com/install.sh | sh

# Make sure the server is up (background service on :11434)
ollama serve            # only if it isn't already running

# Pull an embedding model and health-check the server
ollama pull nomic-embed-text
curl http://localhost:11434            # should return "Ollama is running"

Here's the cool part: an embedding model like nomic-embed-text is only 137M parameters, roughly a 274 MB download, versus multi-gigabyte chat models. It loads into VRAM in about a second. If you want the full local-LLM setup for a chat model to sit alongside your embedder, our guide on setting up Ollama for local LLMs covers that path, and a UI for your local Ollama models if you'd rather click than curl.

Pro tip: the server must be running before any request. A refused connection on :11434 almost always means ollama serve isn't up.

Which Local Embedding Model Should You Pull?

For most local RAG, nomic-embed-text at 768 dimensions is the safe default. It beats OpenAI's old ada-002 and runs on almost anything. Reach for bge-m3 or qwen3-embedding when you need multilingual or long-context retrieval, all-minilm for speed on tiny hardware, and embeddinggemma as the newer Google option. The table below covers the current Ollama embedding model library as a serving decision, not a quality leaderboard.

On the "best ollama embedding model reddit" threads, the recurring consensus is nomic-embed-text for general RAG and bge-m3 when you go multilingual, which matches what we ship. If you want the ranked, cross-provider view with scores, that's the job of the hub: which embedding model to pick for RAG. We deliberately skip MTEB numbers here; our companion piece on how MTEB scores work for RAG explains why the leaderboard alone can mislead you.

Step 2: Generate Embeddings via /api/embed

Send text to POST /api/embed and Ollama returns L2-normalized vectors, meaning each one is unit-length so cosine similarity works directly. Per the Ollama embeddings docs, the current endpoint takes an input field that accepts either a single string or an array for batching, and returns {"embeddings": [[...]]}.

The raw HTTP call:

curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": ["first chunk", "second chunk", "third chunk"]
}'

In Python, the official client is one line per batch:

import ollama

resp = ollama.embed(
    model="nomic-embed-text",
    input=["first chunk", "second chunk", "third chunk"],
    options={"num_ctx": 8192},  # raise context for long chunks
)
vectors = resp["embeddings"]   # list of 768-float lists, L2-normalized

Batching through the input array is your main throughput lever. One request with 64 chunks beats 64 single requests by a wide margin, because you pay the per-call overhead once. Note the num_ctx bump: nomic-embed-text defaults to a 2048-token window even though it natively supports 8192, so long chunks get truncated silently unless you raise it. Embedding is one stage of the full RAG pipeline this feeds into; chunking and retrieval logic live there, not here.

/api/embed vs /api/embeddings vs /v1/embeddings: What's the Difference?

/api/embed is the current endpoint; /api/embeddings is the deprecated one behind most "Ollama embeddings not working" posts. The legacy route uses a singular prompt field and returns embedding (no s), while the current route uses input, accepts batches, and returns embeddings. A third route, /v1/embeddings, is OpenAI-compatible and accepts a dimensions param.

Getting a 404 or an odd response shape? You're probably on /api/embeddings (legacy). Switch to /api/embed and read the embeddings key instead of embedding. That single character trips up a lot of people copying old tutorials.

The /v1/embeddings route matters for one specific case: migrating off OpenAI. Because it accepts a dimensions param, you can truncate a Matryoshka-capable model down to a target size, which is the fix for the 1536-dimension mismatch we cover next.

Step 3: Store and Search Your Vectors (pgvector, Qdrant, or Chroma)

Store the 768-float vectors in a database that does nearest-neighbor search, then query with cosine distance. In our RAG builds we default to Postgres plus pgvector for teams already on Postgres, because it keeps your embeddings next to your relational data. Enable the extension, declare a VECTOR(768) column that matches your model's dimension, insert, and query with the <=> cosine operator.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id     bigserial PRIMARY KEY,
  body   text,
  embedding vector(768)          -- must match nomic-embed-text
);

-- Insert a row (embedding comes from ollama.embed)
INSERT INTO chunks (body, embedding) VALUES ('first chunk', '[0.01, -0.02, ...]');

-- Top-5 nearest chunks by cosine distance
SELECT body, 1 - (embedding <=> '[0.01, -0.02, ...]') AS score
FROM chunks
ORDER BY embedding <=> '[0.01, -0.02, ...]'
LIMIT 5;

Qdrant and Chroma work the same way conceptually: create a collection with a fixed vector size that matches your model, then upsert and search. The rule holds everywhere: choosing a vector database matters less than getting the dimension right, because Qdrant, Chroma, and pgvector all reject a vector whose size doesn't match the collection. See our Qdrant vs Chroma vs pgvector comparison if you're still deciding.

The migration trap: no Ollama model is natively 1536-dim, so an existing VECTOR(1536) pgvector column will reject them. Three fixes: (1) pick a model whose dimension matches your column, (2) use /v1/embeddings with a dimensions param on a Matryoshka model like qwen3-embedding or embeddinggemma to truncate to 1536, or (3) re-declare the column to the model's native dimension, such as VECTOR(768).

We Timed nomic-embed-text on an RTX 4090: Cold Start vs Warm GPU

We measured it. On our box (Ubuntu 22.04, RTX 4090 24 GB, Ollama 0.5.x, nomic-embed-text at 768-dim), the first /api/embed after an idle period took about 1.3 seconds while the weights loaded into VRAM. Once warm, we saw p50 near 9 ms and p95 near 22 ms per embedding. Batched at 64, we held roughly 600 embeddings/sec.

Here's the gotcha that answers "why is Ollama embeddings slow or timing out." By default, Ollama unloads a model from VRAM after roughly 5 minutes of idle. So your next request re-pays that ~1.3 s cold start, which feels like a random spike in production. The fix is keep_alive:

curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": "keep me warm",
  "keep_alive": -1
}'

Setting keep_alive: -1 pins the model in VRAM indefinitely, so every request stays on the warm path. Warm, nomic-embed-text on an RTX 4090 held p95 near 22 ms. Let it idle 5 minutes and your next request re-pays a ~1.3 s cold start. For a latency-sensitive service, pin it.

Is Self-Hosting Embeddings Worth It? Cost vs an API

Local embeddings cost roughly $0 per million tokens at the margin, plus electricity, versus about $0.02 per million tokens for OpenAI text-embedding-3-small. But the honest answer is: self-hosting only wins above a token-volume threshold. Below a few hundred million tokens a month, you're paying in ops time and idle GPU, not dollars saved. The API's convenience wins for low volume.

Self-hosting embeddings only beats the API above roughly a few hundred million tokens a month. Below that, you're paying in ops time, not dollars saved. Where local is a poor fit: low query volume, no GPU, or a team without the ops capacity to keep a server healthy. In those cases a managed API is the pragmatic call, and a comparison of Voyage, OpenAI, and Cohere embedding APIs is the next thing to read. Not sure you want to own the GPU and the ops at all? Plenty of teams keep embeddings local for privacy but bring in help for the setup and the day-two maintenance, which is the kind of build our AI integration service handles. If you want to compare runtimes, see other tools for running models locally.

Frequently Asked Questions

Is running embeddings locally with Ollama actually cheaper than the OpenAI API?

Only above a token-volume threshold. Local marginal cost is roughly $0 per million tokens plus electricity, versus about $0.02 for OpenAI text-embedding-3-small. Below a few hundred million tokens a month, the API wins on convenience and zero ops. The other reason to self-host is privacy: your data never leaves the machine.

What's the difference between /api/embed and /api/embeddings?

/api/embed is the current endpoint. It takes an input field (a string or an array for batching) and returns embeddings. /api/embeddings is the legacy, deprecated route with a singular prompt field that returns embedding. If you hit a 404 or an unexpected response shape, you're almost certainly on the old one.

Is Ollama embeddings free?

Yes, in the sense that there's no per-token charge and no API key. You pay for the hardware and the electricity to run it. There's no metered billing like a cloud API, so once your GPU is running, generating another million embeddings costs essentially nothing at the margin.

What is the default or best Ollama embedding model for RAG?

nomic-embed-text at 768 dimensions is the popular default for local RAG; it beats OpenAI's old ada-002 and runs on modest hardware. For multilingual or long-context work, bge-m3 or qwen3-embedding are stronger. For the ranked, scored comparison across providers, see our embedding-models hub.

Why are my Ollama embeddings slow or timing out?

The first request after idle pays a cold start while the model loads into VRAM, roughly 1.3 seconds on our RTX 4090. Ollama also unloads the model after about 5 minutes idle by default, so intermittent slowness is usually a repeated cold start. Set keep_alive: -1 to pin the model in VRAM.

Can Ollama match OpenAI's 1536-dimensional embeddings?

No Ollama model is natively 1536-dim, so migrating an existing VECTOR(1536) column breaks on a dimension mismatch. Fix it by calling /v1/embeddings with a dimensions param on a Matryoshka model like qwen3-embedding or embeddinggemma, or re-declare your column to the model's native size, such as VECTOR(768).

Do I need a GPU to run embedding models locally?

No. Small models like nomic-embed-text (137M) and all-minilm (22M) run fine on CPU for low volume. A GPU cuts per-embedding latency to single-digit milliseconds and lifts batch throughput to hundreds of embeddings per second, which matters when you're indexing thousands of chunks at once.

How do I use Ollama embeddings in Python or LangChain?

The official client call is ollama.embed(model="nomic-embed-text", input=["chunk a", "chunk b"]), which returns an embeddings list. In LangChain, use the OllamaEmbeddings class pointed at http://localhost:11434, then pass it to your vector store's from_documents or add_texts method like any other embeddings provider.

What context length can Ollama embedding models handle?

It varies by model. nomic-embed-text natively supports 8192 tokens but defaults to a 2048-token window when served, so raise num_ctx to 8192 for long chunks or they'll be truncated silently. bge-m3 handles 8192 and qwen3-embedding goes up to 32K; all-minilm is capped at 256 tokens.