RAG Chunking Strategies: 7 Methods, Ranked by Retrieval Data (2026)

RAG Chunking Strategies: 7 Methods, Ranked by Retrieval Data (2026)

RAG chunking strategies decide what your retriever can find before a single query ever runs. Chroma's July 2024 study ran 472 queries across five corpora on text-embedding-3-large, and the splitter you pick moves recall by about five points: 86.7% for a plain token splitter, 91.7% for the GPT-4o one, with five chunks retrieved per query. Precision swings much harder. Across the full report it runs from 1.5% to 8.0%, which makes your chunk-size choice a cost decision wearing a quality costume, and every page-one guide lists the same seven methods without showing which one retrieves better.

Key Takeaways

• Chunking splits documents before embedding; the split points decide what your retriever can and cannot find.
• In Chroma's July 2024 study of 472 queries, recall ran 86.7% to 91.7% across the measured splitters.
• Precision varies several times more than recall, so chunk size is mostly a token-cost decision.
• Start at 512 tokens with 10% overlap, then tune against your own eval set.

Which RAG Chunking Strategy Should You Use? (Ranked)

For most teams building on flat prose, recursive character chunking at 512 tokens with 10% overlap is the correct default. It respects paragraph and sentence boundaries, costs nothing extra, and in Chroma's 472-query benchmark finished 3.2 points of recall behind the LLM-based splitter. Move away only when your documents have strong structure or your eval set proves otherwise.

Our read: start with recursive character. It trails only the cluster and LLM splitters on recall in Chroma's data, and the LLM splitter's 3.9% precision means you feed the generator roughly twice as much noise per relevant token. Most teams do not have a chunking problem; they have a chunk-size problem they have never measured.

What Does the Data Actually Say About Chunk Size?

The only public head-to-head comparison of RAG chunking strategies is Chroma's technical report "Evaluating Chunking Strategies for Retrieval" (Brandon Smith and Anton Troynikov, published July 3, 2024). They ran 472 queries across 5 corpora (328,208 tokens), embedded everything with OpenAI text-embedding-3-large, and retrieved 5 chunks per query. The rows below come from the report's appendix table for all corpora on text-embedding-3-large at 5 retrieved chunks, so they are directly comparable to each other:

Chroma's main results table, which reports a different retrieval setting, puts the cluster chunker's best precision at 8.0% with 87.3% recall, and stretches the precision range across all its splitters from 1.5% (KamradtSemanticChunker) to 8.0%. Source: Chroma Research, Evaluating Chunking Strategies

Anthropic's "Introducing Contextual Retrieval" (published September 19, 2024) attacks the problem from a different angle. Their baseline top-20 retrieval failure rate was 5.7%; contextual embeddings alone dropped it to 3.7% (35% reduction), contextual BM25 on top brought it to 2.9% (49%), and reranking pushed it to 1.9% (67%). Anthropic does not publish the exact chunk size or overlap used, so treat those as method-level evidence, not size-level. Source: Anthropic, Contextual Retrieval.

Our read: three conclusions from the arithmetic. First, the splitter choice is worth real recall, and Chroma says so plainly: some strategies outperform others by up to 9% in recall. Across its main results table recall runs 83.6% (KamradtSemanticChunker) to 91.9% (LLMSemanticChunker), and inside the retrieve-5 rows above it still spans 86.7% to 91.7%. Precision moves several times further on the same data: 1.5% to 8.0%, a 5.3x spread against recall's 1.1x. So recall is where you pick up a few points, and precision and token cost are where the choice actually bites. Second, the LLM-based splitter buys top recall at worst precision: you pay an LLM call per document and feed the generator more noise. Third, Anthropic's numbers show that enriching chunks with context (5.7% to 3.7%) moved failure rate further than any splitter choice in the Chroma table did. Enrich chunks before you re-tune the splitter. Reranking recovers chunks your splitter mangled, and hybrid search combines BM25 with vector retrieval for the same reason.

Honest limits: both studies use a single embedding model, English-only corpora, and neither is a controlled test of your corpus. Across 472 queries, the gap between the best and worst splitter was about 5 points of recall at 5 retrieved chunks, and a proportionally far larger gap in precision.

Why Does Chunk Size Decide Retrieval Quality?

Chunk size sets the granularity of your retrieval key. A 400-token chunk produces a focused embedding that matches specific queries; a 4,000-token chunk averages across many topics and matches nothing precisely. Small chunks retrieve the exact passage but may fragment an answer across results. Large chunks keep context together but weaken the embedding signal.

The embedding model's context ceiling matters too. If your model caps at 512 input tokens and you feed it 800, the tail is silently truncated. Your embedding represents two-thirds of the chunk. No error logged.

Then the generator side. Liu et al. showed in "Lost in the Middle" (arXiv 2307.03172, 2023) that LLM accuracy drops over 20% when the relevant document sits in the middle of a long context. Retrieving five 1,000-token chunks dumps 5,000 tokens into the prompt, and the answer you need may land in the position the model reads worst. Smaller chunks keep the relevant passage closer to a position the model handles well.

Think of it like a library index. A card that says "Section 4.2, paragraph 3: refund policy" gets you the page. A card that says "everything about commerce in the 20th century" gets you the building. Your embedding is the card. Build a RAG application end to end to see where chunking sits in the pipeline, and read our context engineering guide for how retrieved chunks become prompt tokens. Pinecone's chunking guide frames the same tradeoff from the vector-database side.

Fixed-Size and Recursive Chunking (Start Here)

Fixed-size is the baseline you measure everything against; recursive is what you actually ship.

Fixed-size token chunking

Split every N tokens regardless of content.

def fixed_size_chunks(text: str, size: int = 512, overlap: int = 50) -> list[str]:
    tokens = text.split()  # whitespace proxy; use tiktoken for real token counts
    chunks = []
    step = size - overlap
    for i in range(0, len(tokens), step):
        chunk = " ".join(tokens[i : i + size])
        chunks.append(chunk)
        if i + size >= len(tokens):
            break
    return chunks

Right answer for: flat prose with no heading structure, quick prototypes, and any baseline comparison. It is not stupid. It is the control group.

Recursive character chunking

LangChain's RecursiveCharacterTextSplitter splits on a separator hierarchy: first \n\n (paragraphs), then \n (lines), then . (sentences), then (words). Each chunk stays under chunk_size while respecting the largest natural boundary that fits.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""],
    length_function=len,  # swap for tiktoken len for true token counts
)
chunks = splitter.split_text(document)

The separator list is the part every competitor omits. The splitter tries \n\n first and falls through to . only when a paragraph exceeds chunk_size. If your Markdown has headers, add "## " before "\n\n" so sections stay intact.

Chunk overlap arithmetic: at 512 tokens with 50-token overlap, the step is 462. A 10,000-token document produces ceil(10000 / 462) = 22 chunks. Total embedded tokens: 22 x 512 = 11,264, meaning you re-embed roughly 12.6% of the corpus as overlap. That is the storage and API cost of keeping boundary sentences from orphaning.

How Does Semantic Chunking Work, and Is It Worth the Cost?

Semantic chunking embeds every sentence, measures the cosine distance between neighbouring sentence embeddings, and splits where that distance exceeds a percentile threshold (commonly the 95th). Chunks break at topic shifts rather than arbitrary token counts. Greg Kamradt's "5 Levels of Text Splitting" notebook originated this percentile-breakpoint approach, and the Chroma study benchmarks his chunkers by name.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
splitter = SemanticChunker(
    embeddings,
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95,
)
chunks = splitter.split_text(document)

The cost math is the part nobody puts up front. Semantic chunking embeds your corpus twice: once to compute sentence distances and find breakpoints, once to embed the resulting chunks for indexing. At OpenAI's text-embedding-3-large price of $0.13 per 1M tokens, a 10M-token corpus costs $1.30 to index normally and $2.60 with semantic chunking. You pay double before a single query runs.

What does that buy? In Chroma's retrieve-5 rows, the cluster-based semantic chunker hit 89.0% recall and 6.7% precision versus 88.5% and 7.0% for recursive at the same 200-token size. In the main results table the same chunker posts the study's best precision, 8.0%, at 87.3% recall. Half a point of recall either way, and a precision result that flips sign depending on which retrieval setting you read, for double the embedding bill. Our verdict: semantic chunking pays off on topic-diverse corpora (news archives, paper collections) where fixed boundaries routinely split mid-topic. For homogeneous corpora (product docs, one knowledge base), recursive gets you 95% of the quality at half the cost. If you run embedding models locally with Ollama, the double-embedding cost drops to compute time.

Document-Aware Chunking: Markdown, HTML, and Code

Structure-aware splitting uses the document's own boundaries (headings, list items, function definitions) instead of character counts. A Markdown H2 is a semantic boundary a human placed deliberately; a character splitter shreds it.

from langchain_text_splitters import MarkdownHeaderTextSplitter

headers = [("#", "h1"), ("##", "h2"), ("###", "h3")]
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers)
splits = splitter.split_text(markdown_doc)
# Each split carries metadata: {"h1": "...", "h2": "...", "h3": "..."}

For code, the boundaries are AST nodes. LlamaIndex's NodeParsers ship language-aware splitters that break on function and class definitions. The critical detail: keep the import block and enclosing class signature attached to each function chunk. A function body without its imports is unembeddable noise, so prepend both to every chunk and the embedding captures what the function does and what it depends on.

For code RAG specifically: AST-boundary splitting, imports prepended, 256-512 tokens per function, zero overlap.

What About Late, Hierarchical, and Agentic Chunking?

These are the advanced RAG chunking strategies behind the "RAG 2.0" chatter, and all three sit at 1/10 SERP coverage.

Late chunking

Late chunking, introduced by Günther et al. in "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models" (arXiv 2409.04701, September 2024), embeds the full document with a long-context model first, then pools token-level embeddings into chunk vectors, so each chunk carries document-wide context and "it costs $40/month" knows what "it" refers to. The abstract claims superior retrieval across tasks but publishes no headline number we could verify. Weaviate's writeup explains the mechanics and stops short of a controlled comparison too. Evidence status: promising, unquantified.

Hierarchical (parent-child) chunking

Index small chunks (256 tokens) for retrieval; return the parent (1,024 tokens) to the generator. The retriever finds the needle; the generator gets the surrounding haystack. You maintain two index levels and a parent-child mapping. No public benchmark isolates the effect.

LLM-based / agentic chunking

The Chroma study's LLMSemanticChunker uses GPT-4o to decide split points per document: 91.7% recall (highest) and 3.9% precision (lowest). You pay an LLM call per document at indexing time (about $100 for a 10,000-document corpus) and feed the generator more noise. Reserve it for genuinely irregular corpora: legal filings, scanned PDFs with no extractable headings.

Which Chunk Size Fits Your Embedding Model?

Your embedding model's max input tokens is a truncation ceiling, not a recommendation. A model that accepts 8,192 tokens does not embed better at 8,192 than at 512. Quality degrades with dilution long before the ceiling: the model averages meaning across more tokens and the vector drifts toward the corpus centroid. The recommendation column below is Techsy's interpretation, not the vendors' guidance.

Sources: OpenAI embeddings guide, Cohere embed docs, Voyage embeddings docs, BGE model card.

The pattern: models with a hard 512-token ceiling (Cohere v3, BGE) demand chunks well under 512, because truncation is silent. Feed one 600 tokens and the last 88 vanish from the embedding with no error logged. Models with large ceilings (OpenAI, Voyage, Cohere v4) tolerate bigger chunks but do not reward them. A model's max input length is a truncation limit, not a recommendation.

Pair this with our best embedding models for RAG roundup, what an MTEB score actually measures, and Voyage, OpenAI and Cohere embeddings side by side before you commit to a model.

How Do You Chunk Non-English Documents?

Tokenizers are not language-neutral. Petrov et al. showed in "Language Model Tokenizers Introduce Unfairness Between Languages" (arXiv 2305.15425, 2023) that the same text translated across languages can differ in tokenized length by up to 15x. Even character-level and byte-level models show over 4x difference for some language pairs. A 512-token chunk holds far less meaning in Turkish, Arabic, or Japanese than in English.

Here is the same sentence tokenized with tiktoken's cl100k_base encoding (GPT-4's tokenizer):

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
en = "The retrieval system returns relevant documents."
tr = "Erişim sistemi ilgili belgeleri döndürür."
ja = "検索システムは関連文書を返します。"
print(len(enc.encode(en)), len(enc.encode(tr)), len(enc.encode(ja)))
# 7 tokens, 19 tokens, 19 tokens: same meaning, 2.7x token spread

Counts generated with tiktoken cl100k_base, July 30, 2026.

Practical guidance: at a fixed 512-token chunk size, your Turkish and Japanese chunks hold roughly 37% of the meaning your English chunks do, and your Arabic chunks hold about 26%. Chunk by character count or sentence count per language, or raise the token budget proportionally (about 1,400 for Turkish, 2,000 for Arabic). CJK languages have no whitespace word boundaries, so character splitters behave differently. Arabic morphology packs multiple grammatical markers into single tokens, inflating counts further.

A Decision Tree for Picking Your Chunking Strategy

What kind of document?
├── Structured (Markdown / HTML / code)
│   └── Document-aware splitting on headers or AST boundaries
│       ├── Docs site → MarkdownHeaderTextSplitter, 512 tokens, 0 overlap
│       └── Codebase → AST/function splitter, 256-512 tokens, imports prepended
├── Flat prose (articles, reports, books)
│   └── RecursiveCharacterTextSplitter, 512 tokens, 50 overlap
│       └── Topic-diverse? → try SemanticChunker at 95th percentile
├── Conversational logs (chat, support tickets)
│   └── Split on turn boundaries, group 3-5 turns per chunk, 256 tokens
└── Mixed corpus
    └── Route by MIME type → apply per-type strategy above
        └── Then: how long are expected answers?
            ├── Short (1-2 sentences) → child 256, no parent
            └── Long (multi-paragraph) → hierarchical: child 256, parent 1,024

Three quick prescriptions. Docs chatbot: MarkdownHeaderTextSplitter at 512 tokens, zero overlap, heading path in metadata. Code-search assistant: AST-boundary splitting at 256-512 tokens per function, imports prepended. Mixed enterprise corpus: route by document type at ingestion and store in the vector database you store the chunks in with type metadata for per-type tuning later. That per-document routing is the whole of adaptive chunking for RAG applications.

Tools: LangChain vs LlamaIndex vs Chonkie

We do not sell any of these; the top three ranking pages for this keyword are vendor blogs with product CTAs.

Sources: LangChain docs, LlamaIndex NodeParsers, Chonkie docs.

All three implement the same core algorithms, so pick based on what your pipeline already uses. For the wider RAG tooling stack beyond splitters and Qdrant, Chroma and pgvector compared for storage, see our cluster guides.

How Techsy Approaches Chunking

On client RAG builds, the Techsy team starts at 512 tokens with 10% overlap and does not touch the splitter until we have built a 20-50 question eval set from the client's real support tickets. The eval set comes first; then we change one variable at a time: size, overlap, strategy. No splitter swap without a before-and-after number on the same questions. Get a free consultation for a second set of eyes on your retrieval pipeline.

Frequently Asked Questions

What is chunking in RAG?

Chunking is the preprocessing step that splits documents into smaller segments before embedding, so the retriever can match queries against focused passages rather than entire files. The split points determine what your system can and cannot find at query time.

What is the best chunking strategy for RAG?

For most production systems on general documents, recursive character chunking at 512 tokens with 10% overlap is the strongest default. In Chroma's 472-query study (July 2024), it scored 88.5% recall, within 3.2 points of the most expensive LLM-based method, at zero additional cost.

What is the optimal chunk size for RAG?

Start at 512 tokens. Move down to 256 if your embedding model caps at 512 input tokens (Cohere v3, BGE) or if your queries expect single-sentence answers. Move up to 1,024 only if your eval set shows multi-paragraph answers being fragmented. Always measure against your own questions.

How much chunk overlap should I use?

10-20% of chunk size (50-100 tokens at 512). Overlap prevents boundary sentences from orphaning: a fact split across two chunks appears complete in at least one. Beyond 20%, you re-embed too much of the corpus for diminishing returns. Most teams land on 10% and never revisit it.

Is semantic chunking better than fixed-size chunking?

Marginally, and at double the embedding cost. Chroma's July 2024 benchmark showed the cluster-based semantic chunker at 89.0% recall and 6.7% precision versus 88.5% recall and 7.0% precision for recursive at the same token size, with its 8.0% best-precision result coming from a different retrieval setting. Worth it for topic-diverse corpora; hard to justify for homogeneous document sets.

Does chunk size depend on the embedding model?

Yes. Models with a 512-token input ceiling (BGE, Cohere v3) require chunks well under 512 because truncation is silent. Models with 8,192+ ceilings tolerate larger chunks but do not reward them; embedding quality degrades with dilution before the ceiling. See the pairing table above for per-model starting points.

How do I chunk code for a RAG system?

Split on AST boundaries (function and class definitions) rather than token counts. Keep each chunk at 256-512 tokens per function, prepend the file's import block and enclosing class signature, and use zero overlap since functions are self-contained units. LlamaIndex's CodeSplitter and LangChain's language-aware splitters both handle this.

What is late chunking?

Late chunking embeds the full document with a long-context model first, then pools token-level embeddings into chunk vectors. Each chunk embedding carries document-wide context, solving the "what does 'it' refer to?" problem. Introduced by Günther et al. (arXiv 2409.04701, September 2024). No public head-to-head benchmark quantifies the gain yet.

How do I chunk documents in languages other than English?

Token counts are not language-neutral. The same sentence took 2.7x more tokens in Turkish and Japanese than in English, and 3.9x in Arabic (tiktoken cl100k_base). A fixed 512-token budget silently gives non-English chunks less meaning. Chunk by character or sentence count per language, or raise the budget proportionally.

How do I know if my chunking is actually working?

Build a 20-50 question eval set from real user queries before touching the splitter. Score hit@5 and MRR against your current chunks. Change one variable (size, overlap, strategy), re-run, compare. Without an eval set, you are tuning by feel. Twenty questions is enough to start.

The Bottom Line

• Start with recursive character chunking at 512 tokens, 10% overlap. Correct default for flat prose.
• Across the four splitter families anyone has measured, Chroma's 472 queries move recall about 5 points and precision several times more. Tune for precision and cost first.
• Match chunk size to your embedding model's input ceiling. A 512-token-ceiling model demands chunks under 512.
• Enrich chunks with context (Anthropic's 5.7% to 3.7% failure-rate drop) before re-tuning the splitter.
• Build the eval set first. Every splitter decision without a before-and-after number is a guess.

For the full pipeline around your chunking choice, see build a RAG application end to end. Still deciding between retrieval and fine-tuning? RAG or fine-tuning breaks down when each wins.