Multi-Turn LLM Evaluation: 5 Metrics, 3 Frameworks, 1 Workflow

Multi-Turn LLM Evaluation: 5 Metrics, 3 Frameworks, 1 Workflow

Multi-turn LLM evaluation is the only way to catch the turn-8 amnesia bug: the user gave their order number at turn 3, and the bot asks for it again. Every single turn passed in isolation; the conversation still failed. DeepEval 4.0 and RAGAS 0.4 shipped dedicated conversational eval APIs for exactly this, and after two eval incidents in our own pipeline at Techsy, here are the five metrics, three frameworks, and one workflow to start with.

Key Takeaways

• Multi-turn evaluation scores whole conversations, not isolated input-output pairs.
• Models topping single-turn benchmarks degrade measurably across conversation turns.
• Start with four metrics: completeness, knowledge retention, role adherence, turn relevancy.
• DeepEval, RAGAS, and Langfuse solve multi-turn eval differently; the framework table below compares them.

Why Do Single-Turn Scores Lie to You?

Single-turn evals score one input-output pair at a time, so they cannot see failures that only appear across turns: forgetting, contradiction, drift. A model can post a strong benchmark score and still lose the thread of a live conversation. Laban et al. document this in LLMs Get Lost In Multi-Turn Conversation, 353 citations: performance degrades in multi-turn settings even when single-turn results look healthy.

The core problem is non-determinism: the nth response depends on all n-1 prior turns, so identical prompts behave differently based on history. A dataset of isolated pairs never exercises that dependency. The arXiv survey Evaluating LLM-based Agents for Multi-Turn Conversations, a PRISMA review of roughly 250 sources, splits the field into what to evaluate (context management, planning, coherence) and how (metrics, LLM judges, human review). Both axes are absent from a single-turn suite.

None of this makes your single-turn stack useless. If you run single-turn metrics like BLEU, ROUGE, and G-Eval, keep them for what they measure well: format compliance, toxicity, factual recall on a fixed prompt. Just stop reading them as a health check for the conversation your users touch.

Our interpretation of those studies, in one line:

Single-turn evals measure the answer, multi-turn evaluation measures the conversation, and a model that aces turn one can be lost by turn five.

What Is Multi-Turn LLM Evaluation? The Two Evaluation Modes

Multi-turn LLM evaluation is the practice of scoring a whole conversation, or windows within it, instead of isolated prompt-response pairs. It asks whether the model kept context, stayed in role, and resolved the user's problem across turns. Two modes do the work: conversation-level scoring and sliding-window turn-level scoring, and most teams run both.

Conversation-level scoring hands the judge the full transcript and asks one question: was this conversation successful? It catches premature closure and unresolved loops, since only the whole thread reveals the user never got their refund. Its weakness is granularity: "failed" on a 12-turn thread does not say where things broke.

Sliding-window turn-level scoring moves a window of N turns across the transcript, one verdict per window. A window of 3 over a 10-turn conversation yields 8 verdicts tied to regions of the chat, so "failed" comes with coordinates: the break happened in turns 6 through 8. The diagram at the top of this post shows both modes on one thread: a bracket for the conversation verdict, a sliding frame for per-window verdicts.

Use conversation-level scoring as the gate, windowed scoring to localize failures when it trips. DeepEval's multi-turn evaluation guide frames the unit of work as a scenario rather than an input-output pair (its ConversationalGolden type): you are testing a situation, not a question.

Illustrative example (synthetic; shows the mechanics, not a real run): a sliding window of 3 across an 8-turn return-request chat.

Turn 1  user:      I want to return an order that arrived damaged.
Turn 2  assistant: Sorry about that. Can you share the order number?
Turn 3  user:      It's #4471.
Turn 4  assistant: Got it. Damaged on arrival, or after use?
Turn 5  user:      On arrival. The screen was cracked.
Turn 6  assistant: Understood. Replacement or refund?
Turn 7  user:      Refund. How long does that take?
Turn 8  assistant: 3-5 business days. Can you share the order number again?

Conversation-level verdict: fail. Five of six windows passed, and the thread still broke on knowledge retention, exactly the failure a single-turn suite never surfaces.

Which Multi-Turn Metrics Matter? The 5 That Do

Run four metrics first: conversation completeness, knowledge retention, role adherence, and turn relevancy. Add a fifth, a custom criterion (G-Eval in DeepEval, AspectCritic in RAGAS), for whatever your product cannot get wrong. The first four transfer between projects; the fifth is where your failure modes live.

1. Conversation completeness. Did the user's goal get resolved, or did the bot declare victory early? Your premature-closure detector.
2. Knowledge retention. Does the model remember facts stated earlier in the thread? The turn-8 amnesia bug is a knowledge-retention failure.
3. Role adherence. Does the assistant stay inside its persona and refuse out-of-scope requests? Critical with a compliance boundary.
4. Turn relevancy. Is each response on-topic given the preceding turns? Catches drift and loops.
5. A custom criterion. One plain-English rule for your domain: "never quote a price that differs from the price list." DeepEval implements this as ConversationalGEval; RAGAS as AspectCritic.

The DeepEval metrics guide defines each with runnable classes, but the concepts are framework-neutral: the table holds even if you hand-roll your judge.

A custom criterion reads like a sentence:

criterion "price_accuracy":
  question: Does the assistant quote prices matching the official
            list, and self-correct when the user flags a mismatch?
  scale: 0 (wrong, no correction) to 1 (correct throughout)
  verdict: pass if score >= 0.5

The same rule as real DeepEval code:

from deepeval.metrics import ConversationalGEval
from deepeval.test_case import LLMTestCaseParams

price_accuracy = ConversationalGEval(
    name="Price Accuracy",
    criteria=(
        "Does the assistant quote prices that match the official "
        "price list, and correct itself immediately when the user "
        "points out a discrepancy?"
    ),
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
    ],
    threshold=0.5,
)

DeepEval vs RAGAS vs Langfuse: Which Framework Fits?

All three evaluate multi-turn conversations, but their unit of evaluation differs: DeepEval simulates scenarios offline, RAGAS scores aspects of conversations you already have, and Langfuse evaluates real production traces. Pick by where your conversations come from, not feature count.

Framework-neutral logic first, so the vendor code below is portable:

for scenario in scenario_set:
    transcript = run_chatbot(scenario, max_turns=10)
    for window in sliding_windows(transcript, 3):
        scores.append(judge(window, criteria))
    scores.append(judge(transcript, completeness))
fail_if(mean(scores) < baseline - tolerance)

DeepEval: scenarios and a batteries-included simulator

DeepEval is the only one with a first-class conversation simulator: describe a scenario and a persona, and it plays the user against your bot. Its multi-turn guide is the canonical reference for the scenario-not-pairs pattern. Confident AI sells the hosted dashboard; our Confident AI review covers what the paid layer adds.

from deepeval.dataset import ConversationalGolden
from deepeval.synthesizer import ConversationSimulator
from deepeval.metrics import (
    ConversationCompletenessMetric,
    KnowledgeRetentionMetric,
)
from deepeval import evaluate

scenario = ConversationalGolden(
    additional_context="Customer wants to return a damaged order",
    user_persona="Impatient customer, second contact this week",
)
simulator = ConversationSimulator(model="gpt-4o-mini", max_turns=10)
test_case = simulator.simulate(scenario, your_chatbot_fn)

evaluate(
    test_cases=[test_case],
    metrics=[
        ConversationCompletenessMetric(threshold=0.7),
        KnowledgeRetentionMetric(threshold=0.7),
    ],
)

RAGAS: error-analysis-driven, aspect-by-aspect

RAGAS starts from conversations you already have and scores them aspect by aspect. Its multi-turn how-to pairs with manual error analysis: read failing chats, write an AspectCritic per failure mode, score.

from ragas.dataset_schema import MultiTurnSample
from ragas.metrics import AspectCritic
from ragas.llms import llm_factory

user_input = [
    {"role": "user", "content": "Can I return a damaged order?"},
    {"role": "assistant", "content": "Yes, within 30 days."},
    {"role": "user", "content": "It arrived broken. Do I pay shipping?"},
    {"role": "assistant", "content": "No, we cover it."},
]
sample = MultiTurnSample(user_input=user_input)

critic = AspectCritic(
    name="policy_consistency",
    definition="Does the assistant stay consistent with the stated return policy across all turns? Answer yes or no.",
    llm=llm_factory("gpt-4o-mini"),
)
score = await critic.multi_turn_ascore(sample)  # binary 0 or 1

Langfuse: N+1 evaluation on real traces

Langfuse takes the opposite path: tracer first. Its N+1 cookbook evaluates each turn's trace plus the conversation as a whole, on production traffic rather than simulations. If you are still choosing the observability layer, our Langfuse vs LangSmith comparison covers that decision.

Our verdict, no fence-sitting: for a new chatbot project, start with DeepEval. The simulator lets you gate regressions before you have production traffic, when you most need tests. Add Langfuse once real threads exist; reach for RAGAS when your team prefers reading failing conversations and codifying what they find.

How Do You Go From Error Analysis to Automation?

You sequence it. Read 20-30 real conversations, label failure modes by hand, write binary pass/fail checks for the obvious ones, automate those, and only then add LLM-judged metrics for the subjective residue. Hamel Husain argues for exactly this order: manual error analysis and binary decisions first, because a check you can explain beats a score you cannot.

Binary before judge: the sequencing that saved us

This is not a chatbot benchmark we ran; it is our interpretation of the same pattern inside our own content pipeline, which runs eval-gated regression checks on every prompt and tooling change. Two incidents proved the sequencing for us.

On 2026-06-13, a republish bug minted new localized slugs and shipped 54 duplicate live documents. We found and unpublished them on 2026-07-05 (backup at techsy.io/seo-reports/2026-07-05/deleted_docs_backup.json). The fix was not a smarter model; it was a deterministic pre-publish check: resolve the existing document by canonical post plus language before any create. A binary gate.

Second incident: translator LLMs occasionally emit ASCII instead of Unicode, turning "karşılaştırma" into "karsilastirma." No judge needed; a grep gate catches it:

grep -cP '[çşğüöıİŞÇĞÜÖ]' file.md   # must be > 0

Both were caught by checks that cost fractions of a cent and print exactly why they failed. Map that to multi-turn evals: "did the bot re-ask for a field the user already gave?" is a string match against the transcript, not a judge call. Run cheap deterministic gates first; they catch the ugly failures before your expensive judge ever runs.

When an LLM judge is actually the right tool

Judges earn their token cost on criteria you cannot reduce to a rule: "was the tone appropriately apologetic?", "did the resolution fit the situation?" If you can write an assertion, write an assertion. A rubric full of judgment calls is judge territory.

The line we keep coming back to:

Start with binary pass/fail checks you can explain to a teammate, then add LLM judges only for what you cannot reduce to a rule.

How Do You Simulate Conversations at Scale, and What Does Judging Cost?

Simulate from scenarios, not exported logs. Scenarios test what could happen; logs only show what your current system already allowed. DeepEval's guidance cautions that historical conversations were shaped by the system that produced them, so benchmarking against them bakes in the status quo.

Scenarios, not transcripts

Write each scenario as goal plus persona: "impatient customer returning a damaged order," "user who changes their mind mid-booking." Set a max-turns limit (10 is sane) and a stopping condition: goal reached, user abandons, or cap. DeepEval recommends at least 20 diverse scenarios across primary use cases, edge cases, and failure-prone situations; below that, your suite measures anecdotes.

Adversarial personas

Include personas that try to break the bot: an angry user who escalates, a confused user who contradicts themselves, an injection user who slips instructions into turn 4. Multi-turn injection is its own discipline; our LLM guardrails guide covers the defensive layer that pairs with these tests, and Langfuse's simulation cookbook shows the user-simulator loop.

What 100 evaluated conversations cost

Every figure below is an estimate from stated token counts and public pricing, not a measurement we ran. The arithmetic is the point: swap in your own numbers.

Under a dollar for 100 fully judged conversations. A pricier judge moves this 10-50x, and the tactics in our reduce LLM API costs guide apply: cache the criteria text, batch windows, use the cheap model for binary gates.

A 6-Step Multi-Turn Eval Workflow

The loop runs like this: define scenarios from real failures, pick four core metrics plus one custom, simulate at least 20 scenarios, baseline the current version, gate regressions in CI, and feed production failures back into the scenario set.

1. Define scenarios from failures. Read 20-30 transcripts (or, pre-launch, write them from support tickets). Each scenario gets a goal, a persona, and a max-turns cap. Owner: you and Hamel's error-analysis-first method.
2. Pick four metrics, one custom. Completeness, knowledge retention, role adherence, turn relevancy, and one ConversationalGEval or AspectCritic for your domain's expensive mistake.
3. Simulate. Run at least 20 scenarios including the adversarial set. Owner: DeepEval's ConversationSimulator, or Langfuse's simulation cookbook.
4. Baseline the current version. Record per-metric means over 3 runs, since models are non-deterministic and a single run is noise. Owner: your eval script, results committed to the repo.
5. Gate regressions in CI. Set a threshold per metric and fail the build on regression beyond a tolerance:

# ci/multi-turn-eval-gate.sh
set -euo pipefail
python eval/run_multi_turn.py --scenarios eval/scenarios.yaml --out results.json
SCORE=$(jq -r '.aggregate.completeness' results.json)
BASELINE=0.82
TOLERANCE=0.03
if (( $(echo "$SCORE < $BASELINE - $TOLERANCE" | bc -l) )); then
  echo "FAIL: completeness $SCORE below baseline $BASELINE"
  exit 1
fi

1. Monitor production threads. Group live traces by thread, evaluate asynchronously, and turn every failing thread into a new scenario. Owner: Langfuse or your tracer; our guides on evaluating AI agents in production and AI observability cover the monitoring half.

The suite is never finished: step 6 feeds step 1, and the scenario set grows with every production failure you catch.

How Do You Evaluate Tone Across Languages?

A role-adherence metric tuned on English data will pass a Turkish or Japanese transcript a native speaker finds rude, because politeness register is language-specific. Your English rubric has no words for it. The fix: one aspect criterion per register expectation, written per language, not one global tone metric.

One criterion per register

Our interpretation of the RAGAS AspectCritic pattern, extended from running a 23-language pipeline, not a published test result:

English:  "Is the assistant's tone friendly but professional?"
Turkish:  "Does the assistant use formal 'siz' address consistently,
           and avoid casual verb forms with an upset customer?"
Japanese: "Does the assistant keep keigo (polite form) throughout,
           including the apology at the resolution turn?"

Each criterion is a separate binary critic on the same transcript. We have not published cross-language tone scores, and would not trust an article that prints them without the rubric. From the pipeline work: failures cluster at apology and escalation turns, where register collapses first.

Frequently Asked Questions

What is a multi-turn conversation LLM?

A language model whose nth response depends on all prior turns, not just the latest prompt. It conditions on the full thread, so behavior changes with conversation history. That context-dependence is what single-turn tests cannot exercise and multi-turn evaluation exists to score.

What does LLM evaluation mean?

Measuring output quality against defined criteria, automatically and repeatably, instead of by feel. Single-turn evaluation scores isolated prompt-response pairs against metrics like BLEU or an LLM judge. Multi-turn evaluation extends that to whole conversations, scoring context retention and goal completion across turns rather than per prompt.

How do you benchmark multi-turn LLM performance?

Build at least 20 scenarios with goals and personas, simulate them against the model, and score with conversation-level metrics plus sliding-window checks. Record baselines over multiple runs to absorb non-determinism, then compare each new version against the baseline in CI. Production traces extend the benchmark later.

What are the best ways to evaluate an LLM?

Sequence it: manual error analysis first, then binary pass/fail gates for everything reducible to a rule, then LLM-as-a-judge for subjective criteria like tone and resolution quality. Binary checks are cheaper, debuggable, and do not drift; judges belong on criteria that genuinely require judgment, after the cheap gates pass.

Which multi-turn evaluation metrics should I start with?

Conversation completeness, turn relevancy, and knowledge retention; they catch the most common failures (unresolved goals, drift, forgetting) in any chat product. Add role adherence if your bot has a compliance boundary, then one custom G-Eval or AspectCritic criterion for the mistake your business cannot afford.

How much does LLM-as-a-judge cost per conversation?

With a sliding window of 5 over 10 turns plus one conversation-level call, you make 7 judge calls per conversation. At roughly 2,000 input tokens per call on GPT-4o-mini, our shown-math estimate works out to about $0.29 per 100 conversations. Premium judge models raise that 10-50x.

DeepEval vs RAGAS for multi-turn evaluation: which should I pick?

DeepEval if you want offline regression tests with a built-in conversation simulator, especially before you have production traffic. RAGAS if your workflow starts from reading real failing conversations and codifying each failure mode as an AspectCritic. One common split: DeepEval in CI, RAGAS-style critics on production logs.

How many scenarios do I need for a multi-turn eval suite?

At least 20, covering primary use cases, edge cases, and failure-prone situations; that threshold comes from DeepEval's published guidance and matches our experience. Below 20, pass rates swing on which scenarios happened to be included. Grow the set with every production failure.

Can I run multi-turn evaluation in CI/CD?

Yes. Keep a fixed scenario set in the repo, run it on every prompt or model change, and fail the build when a metric regresses past tolerance against the baseline. Because models are non-deterministic, compare means over 3 runs with a tolerance (we use 0.03), not exact thresholds.

How do I evaluate multi-turn conversations in production?

Group traces by conversation thread, score each thread asynchronously so evaluation never blocks a response, and route failing threads into a review queue. Every confirmed failure becomes a new scenario in your offline suite, closing the loop between monitoring and regression tests.

The Short Version

• Single-turn scores cannot see conversational failures; research shows models degrading across turns despite healthy benchmarks.
• Run conversation-level scoring as your gate and sliding-window scoring to localize breaks.
• Four core metrics plus one custom criterion cover most chat products; binary checks before judges, always.
• DeepEval for simulated regression tests, RAGAS for error-analysis-driven critics, Langfuse for production traces.
• Judge costs are small (under a dollar per 100 conversations on a mini model); cost is rarely the blocker.

For the wider tool landscape, we ranked the full field in our best LLM evaluation tools roundup. And if you would rather build the eval pipeline with someone, get a free consultation with the Techsy team.