Best Open-Source LLM Evaluation Frameworks in 2026 (One Isn't Actually Open Source)

Best Open-Source LLM Evaluation Frameworks in 2026 (One Isn't Actually Open Source)

Line 1 of the LICENSE file in the Arize Phoenix repo reads "Elastic License 2.0 (ELv2)". Not Apache. Not MIT. One heavily recommended open source LLM evaluation framework isn't open source by the OSI definition, and nearly every page ranking for this query repeats the claim anyway. So did one of ours, until today. On 2026-08-04 we read the license file and default-branch commit log of eight frameworks by hand, plus three more the top pages still recommend, then installed six and ran the same 10 cases through each. We don't sell an eval framework, so no verdict below is protecting a product.

Key Takeaways

• Arize Phoenix ships under Elastic License 2.0, which the OSI does not approve as open source.
• UpTrain's last commit to main was 2024-07-29. Don't start a new project on it.
• pip install promptfoo gets a third-party wrapper. The real project ships on npm.
• Ragas has no commits since 2026-02-24 and moved GitHub orgs to vibrantlabsai.

Which Open-Source LLM Evaluation Framework Should You Install in 2026?

Pick by constraint, not by rank. For pytest-shaped assertions inside an existing test suite, install DeepEval. For a YAML config and a CLI that fits any language stack, install promptfoo. For the cleanest good-versus-bad separation we measured, install Opik. All three are Apache-2.0 or MIT.

Here is the audit. Eight frameworks in scope, plus three more that the top-ranking pages for this query still recommend.

Dates are the last commit on each project's default branch as of 2026-08-04. GitHub's repo page shows the last push to any branch, which is later for two projects here: UpTrain 2024-08-18 and Deepchecks 2025-12-28. Neither repo is archived.

The switch-cost column is the one people skip and then regret. Scores are just numbers, so moving between DeepEval, Ragas, Opik and phoenix-evals mostly means rewriting a loop. Moving off promptfoo or Inspect AI means rewriting a config or task format with no equivalent elsewhere, and moving off Evidently means auditing every threshold you wrote, because its scale runs the other way. Two of the frameworks the top-ranking pages still recommend haven't shipped a release since 2024.

Want tiers, hosted platforms and a straight ordering instead? That's a different job, and we already did it in our ranked comparison of LLM evaluation tools including paid platforms.

The Eight LLM Evaluation Frameworks, Grouped by How You Install Them

Installation shape is what you have to live with, so that's the grouping.

Python libraries you import into tests

DeepEval (pip install deepeval, Apache-2.0) wraps LLM metrics in pytest-shaped assertions: build an LLMTestCase, hand it to assert_test, and the test fails below your threshold. Best at putting a quality gate next to the unit tests a team already runs. Pick this if your evals belong in the same CI job as everything else.

One disclosure, stated once: DeepEval is built by Confident AI, which is a paid partner on two other posts on this site, including the ranked comparison this page links to. It gets no special treatment here, and every DeepEval link on this page points at the GitHub repo.

Ragas (pip install ragas, Apache-2.0) is the RAG-specific option: evaluate() takes question, context and answer rows and returns per-metric scores asynchronously. Best at retrieval-quality measurement inside a Python pipeline. Its repo moved from explodinggradients to vibrantlabsai, its last release was v0.4.3 on 2026-01-13, and there are no commits since 2026-02-24. Pick this if RAG metrics are the whole job and a quiet repo is acceptable, and see the wider RAG tooling stack.

Opik (pip install opik, Apache-2.0, from Comet) ships metrics you can call on their own. Set OPIK_TRACK_DISABLE=true and AnswerRelevance().score() runs with no account, no local server and no config file, which the product framing doesn't advertise. Best at getting a real score in the fewest lines. Pick this if you want metrics now and the platform maybe later.

Evidently (pip install evidently, Apache-2.0) treats evals as descriptors over a dataset and writes an HTML report as a side effect. Best at batch reporting across many rows rather than a binary gate. Its LLM scores are inverted: 1.0 means unfaithful. Pick this if what you owe someone is a shareable report, not a red build.

Giskard (pip install giskard, Apache-2.0) scans a model for vulnerabilities instead of scoring a dataset you wrote. The PyPI package resolves the v2 line, and the project's own README states that v2 "is no longer actively maintained". Best at automated red-team style scans. Pick this if you want vulnerabilities found for you rather than LLM-as-a-judge metrics you define yourself.

CLI-and-config tools you run against a YAML file

promptfoo (npm install promptfoo, MIT) is a CLI that reads a YAML file: declare providers, test cases and assertions, run npx promptfoo eval, and get pass/fail per case plus a local results UI. Best at evaluating prompts when your app isn't written in Python. Pick this if your quality gate should be a config file that a non-Python teammate can edit.

Harness-class and platform-bundled

Inspect AI (pip install inspect-ai, MIT) comes from the UK AI Safety Institute and evaluates models against tasks you define in Python, with real solver and scorer abstractions and a run viewer. Best at model-level benchmarking with reproducible task definitions. Pick this if the thing under test is a model rather than your application.

Arize Phoenix (pip install arize-phoenix-evals) gives you prebuilt evaluators such as FaithfulnessEvaluator and CorrectnessEvaluator that return a binary label plus a score. Best at deterministic labels you can gate on without picking a cutoff. Its license is the reason this article has a parenthetical in the title, and that gets its own section next.

Is Arize Phoenix Open Source?

No, not by the definition the Open Source Initiative maintains. Arize Phoenix ships under the Elastic License 2.0 (ELv2). Line 1 of the repo's LICENSE file says so, and PyPI independently declares license: Elastic-2.0 on v19.15.0. The source is readable, forkable and self-hostable. One use is restricted.

The restriction that matters: ELv2 forbids providing the software to third parties as a hosted or managed service. Read that carefully, because it binds far fewer people than it sounds like. If you install arize-phoenix-evals to score your own application, ELv2 never touches you. If you're a consultancy or a platform team packaging Phoenix into an eval service you sell to external customers, it does. That's the whole delta, and the Open Source Definition is what ELv2 fails, specifically the clauses on field-of-use restrictions.

Every page currently ranking for this query files Phoenix under "open source", and so did we. Our own ranked comparison of LLM evaluation tools describes Phoenix as fully open-source, which is wrong, and it's being corrected. Phoenix is source-available, not open source, and the distinction only bites if you plan to sell it as a service. If tracing rather than scoring is what you actually need, that belongs to AI observability platforms, not here.

Which of These Are Still Actively Maintained?

Most of them. Six of the eleven repos we checked took a commit to main on 2026-08-03 or 2026-08-04: DeepEval, promptfoo, Opik, Arize Phoenix, Inspect AI and Giskard. Two have not shipped a release since 2024. One has gone quiet in 2026 after a change of GitHub organization.

Frameworks we wouldn't start a new project on in 2026

UpTrain is dead. Its last commit to main landed on 2024-07-29 and its last release, v0.7.1, was 2024-05-14, which puts it two years cold on either measure. The repo is still there and still Apache-2.0, so nothing stops you, but starting new work on an abandoned evaluation library is a decision you'll be explaining later.

Deepchecks deserves the precise version. It has had no release since 0.19.1 on 2024-12-15, though the repo still receives commits, with the last one on main dated 2025-11-24. People are still working on it; nobody has cut a version in over eighteen months. Neither UpTrain nor Deepchecks is archived on GitHub, and neither has closed to contributions.

Ragas gets dates and nothing else. Last release v0.4.3 on 2026-01-13, no commits since 2026-02-24, and the repo moved from explodinggradients to vibrantlabsai. We found no verifiable account of why the org changed, so we won't invent one. A quiet repo isn't a broken one: Apache-2.0 code that computes a faithfulness score today still computes it next year. The exposure is unpatched dependencies, which is exactly what bit us in testing below.

Other pages on page one of this query still recommend both UpTrain and Deepchecks, with no date attached to the recommendation. A framework with no release since December 2024 is a dependency decision, not a feature decision.

Do You Need an Eval Framework or an Eval Harness?

An application eval framework scores your app's own outputs against your own data. DeepEval, Ragas, promptfoo, Opik, phoenix-evals and Evidently all do that. A model eval harness benchmarks a model against standardized public tasks instead. lm-evaluation-harness and Inspect AI do that. Picking the wrong class is the most expensive mistake on this page.

The failure mode is concrete. Someone wires lm-evaluation-harness up to test their RAG chatbot, gets a set of MMLU scores back, and learns precisely nothing about whether their retriever is returning the right passages. The scores are real. They're measuring the base model, which nobody was worried about.

Inspect AI's shape follows from its provenance: it was built at the UK AI Safety Institute under MIT for evaluating frontier models, so solvers, scorers and tasks are first-class and your application is not a concept it has. That's a good reason to use it for what it's for. If your problem is agents rather than single turns, evaluating agents in production is a different discipline again, and tool-calling servers get their own treatment in our guide to evaluating MCP servers and tools.

What Happened When We Installed Six of Them and Ran the Same 10 Cases

On 2026-08-04 we installed six of these in fresh Python 3.11.14 venvs (plus npm for promptfoo) and scored one identical 10-item RAG set with one judge, openai/gpt-4o-mini through OpenRouter at temperature 0. Seven items were correct. Three were broken in three different ways: one contradicts its context, one invents specifics, one is fluent prose that never answers the question. Each framework ran twice, back to back.

Five of six grounding metrics caught all three defects. The three findings below are the reason this section exists.

Relevance metrics are not quality metrics, and two of them rated a confident lie above a correct answer. Ragas ResponseRelevancy scored the item asserting that HTTP 404 is a 5xx server error at 0.777, above two of the seven correct answers, and the item with invented rate limits at 0.813, above four. promptfoo answer-relevance did the same: 0.800 for the 404 item, a clean pass against a 0.7 threshold, while failing the correct q01 at 0.679. Not a bug. A confident wrong answer addresses the question perfectly. But if relevance is the number on your dashboard, a fluent hallucination looks like your best output.

A faithfulness-only gate misses the irrelevant answer. DeepEval scored the item that never answers the question at 1.000 faithfulness, a clean pass, which is defensible: an answer that asserts nothing about the context contradicts nothing in it. Only relevancy caught it, at 0.000. That is the one grounding miss in the table above. Either metric alone has a hole; the pair covers both.

Binary evaluators were stable at temperature 0. Graded ones weren't. Phoenix and DeepEval moved zero of ten items across two identical runs. Opik moved four, all on AnswerRelevance, on a 0.05 grid; Evidently also moved four. No drift flipped a verdict here, but promptfoo's correct q01 landed at 0.679 then 0.642 against a 0.700 threshold, which is the shape of a flaky CI gate.

Two smaller notes: three of six (DeepEval, Opik, Phoenix) installed and ran clean first time, while Ragas wouldn't import until we pinned langchain-community<0.4. Only promptfoo reported judge token usage, 16,011 assertion tokens in run 1 and 16,010 in run 2.

The limits of this, stated plainly. n = 10 is a smoke test, not a benchmark: it tells you about ergonomics and blind spots, not metric accuracy. One judge model scored everything, and a larger judge would move every number, probably including the two false positives DeepEval and Ragas both produced on the same correct item. Two runs prove drift exists and cannot characterize it. Answers were pre-written, so nothing here exercises generation, tracing or dataset management, which makes promptfoo's 337.9-second install look worse than it deserves. "Best" always means best for a constraint: a CI gate, RAG metrics or a UI each change the answer, as does the split between offline and online evaluation.

The Same Check, Written Three Ways

The fastest way to pick an interface shape is to read the same assertion three times. Here's a grounding check on one item in DeepEval, Ragas and promptfoo, trimmed from the scripts we actually ran. The metric names differ; we link to how LLM-as-a-judge metrics actually work rather than redefine them here.

# DeepEval 4.1.5: pytest-shaped, fails the test below the threshold
from deepeval import assert_test
from deepeval.models import GPTModel
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

judge = GPTModel(
    model="openai/gpt-4o-mini",
    base_url="https://openrouter.ai/api/v1",
    api_key=OPENROUTER_KEY,
)

def test_faithfulness():
    case = LLMTestCase(
        input=question,
        actual_output=answer,
        retrieval_context=[context],
    )
    assert_test(case, [FaithfulnessMetric(threshold=0.7, model=judge)])

# Ragas 0.4.3: note the import path. `from ragas.metrics import Faithfulness`
# raises ImportError in this version; the concrete metric moved.
from ragas import evaluate, EvaluationDataset
from ragas.metrics._faithfulness import Faithfulness
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

judge = LangchainLLMWrapper(
    ChatOpenAI(model="openai/gpt-4o-mini",
               base_url="https://openrouter.ai/api/v1")
)

result = evaluate(
    dataset=EvaluationDataset.from_list(rows),
    metrics=[Faithfulness(llm=judge)],
)

# promptfoo 0.121.20: npm install promptfoo, then npx promptfoo eval
providers:
  - id: echo          # we scored pre-written answers instead of generating them
defaultTest:
  assert:
    - type: context-faithfulness
      threshold: 0.7
tests:
  - vars:
      query: "Is HTTP 404 a client error or a server error?"
      context: "HTTP 404 Not Found is in the 4xx class, which denotes client errors."
      output: "HTTP 404 is a server error in the 5xx class."

The install lines carry more traps than the code does, and every comment below is something that cost us time on 2026-08-04:

# The real promptfoo ships on npm. The PyPI package of the same name is a
# third-party wrapper: https://pypi.org/project/promptfoo/ vs
# https://www.npmjs.com/package/promptfoo
npm install promptfoo

# pip install giskard resolves the v2 line, which the project's own README
# marks as no longer actively maintained.
pip install giskard

# lm-evaluation-harness installs under the package name lm-eval.
pip install lm-eval

# Evidently does not pull openai, and the judge crashes at call time rather
# than import time, after you've already built the dataset.
pip install evidently openai

# Ragas 0.4.3 will not import against langchain-community 0.4.x.
pip install ragas "langchain-community<0.4"

Can You Fail a Build on an Eval Score?

Yes. Every framework here returns a numeric or binary score, and each will exit non-zero when a threshold assertion fails, which is all GitHub Actions needs to turn a build red. Wiring the exit code is the easy part. Choosing a threshold your judge model won't cross by accident is the part that takes a week.

This is the workflow shape we run, pinned to the versions from our 2026-08-04 test:

name: evals
on: [pull_request]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11.14"
      - run: pip install deepeval==4.1.5

      - name: Score the golden set
        env:
          # Pin the judge. A model upgrade mid-quarter moves every score.
          JUDGE_MODEL: openai/gpt-4o-mini
          # Thresholds live in one place, read by the metric constructors.
          EVAL_THRESHOLD: "0.7"
          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
        run: deepeval test run tests/evals/

Two gotchas bite before the threshold does. First, judge calls are network calls: our 10-item DeepEval run took 126.8 seconds because .measure() is sequential, and a 200-item golden set on that code path is a coffee break on every pull request. Every other framework in the test parallelizes by default, which is the biggest single lever on CI wall-clock time.

Second, flakiness. At temperature 0, Opik and Evidently each moved four of ten items between back-to-back runs, and promptfoo's correct answer sat at 0.679 then 0.642 against a 0.700 gate. The mitigations are boring and they work: run a fixed golden dataset that changes only by pull request, pin the judge model, prefer binary evaluators where a label will do, and gate on a delta rather than an absolute floor. That last one matters most for multi-turn evaluation, where one conversation produces many scores that can each wobble.

For scale: LangChain's State of Agent Engineering survey (1,340 responses, fielded 18 November to 2 December 2025, published 12 June 2026) found 89% of organizations have implemented some form of observability for their agents, while just 52.4% run offline evaluations on test sets. Watching is common. Gating is not.

What We'd Install This Week

Four things to take away. Arize Phoenix is source-available under Elastic License 2.0 and is not OSI-approved open source, which changes nothing for most readers and everything if you resell eval tooling. UpTrain is dead, and pages with no date on them still recommend it. Deepchecks hasn't cut a release since December 2024 either, though its repo does still take commits. A grounding metric and a relevance metric each have a hole the other covers, so gate on both. And graded scores drift at temperature 0, so pin your judge and give thresholds room.

If I were starting a new eval suite this week, I'd install DeepEval for the CI gate because assertions belong next to tests (disclosure above), and add Opik's standalone metrics for the cleanest separation we measured. If our stack weren't Python, promptfoo instead, no hesitation. If you'd rather someone else wired the golden set and the workflow, that's a conversation we're happy to have.

Frequently Asked Questions

What is the best open-source LLM evaluation framework?

There isn't one winner, only a best fit per constraint. For a pass/fail gate inside a Python test suite, DeepEval. For a language-agnostic YAML and CLI setup, promptfoo. For RAG retrieval metrics, Ragas, if you can accept a repo with no commits since 2026-02-24. For the cleanest separation between good and bad answers in our 2026-08-04 test, Opik.

Is Arize Phoenix open source?

Not by the Open Source Initiative's definition. Arize Phoenix ships under the Elastic License 2.0, which PyPI declares as license: Elastic-2.0 on v19.15.0 and which line 1 of the repo's LICENSE file states directly. It's source-available: you can read, fork, modify and self-host it. The one restriction is offering the software to third parties as a hosted or managed service.

Is Ragas still maintained?

The verifiable facts, as of 2026-08-04: the last release was v0.4.3 on 2026-01-13, there have been no commits since 2026-02-24, and the repository moved from the explodinggradients organization to vibrantlabsai. The repo is not archived. We found no reliable public explanation for the org change and won't speculate about one. The Apache-2.0 code still runs; the exposure is unpatched dependencies.

Do I need an eval framework or an observability platform?

Both, eventually, but they answer different questions. An eval framework tells you whether a change made your outputs better or worse before you ship it, on a dataset you control. An observability platform tells you what actually happened in production after you shipped. Start with the eval framework if you have a CI pipeline; see AI observability platforms for the production side.

Can I run LLM evals in CI/CD?

Yes. Every framework covered here exits non-zero on a failed threshold assertion, which is all a GitHub Actions job needs. The practical constraints are wall-clock time (judge calls are network calls, and our sequential DeepEval run took 126.8 seconds for 10 items) and judge non-determinism. The workflow shape and mitigations are in the CI section above.

What's the difference between DeepEval and Ragas?

Interface shape and scope, not quality. DeepEval is pytest-shaped and general purpose: you write test cases and assert on metric thresholds, and it covers application outputs of many kinds. Ragas is a RAG-specific library whose evaluate() runs asynchronously over a dataset of question, context and answer rows. DeepEval fits a CI gate more naturally; Ragas goes deeper on retrieval.

Why does pip install promptfoo give me the wrong package?

Because promptfoo is a Node project. The real one is published on npm under MIT and installs with npm install promptfoo. The PyPI package of the same name is a third-party wrapper, not the upstream project, and installing it is a common way to end up debugging a CLI that isn't the one the docs describe.

Is lm-evaluation-harness an LLM evaluation framework?

It's a model evaluation harness, which is a related but different job. lm-evaluation-harness (installed as pip install lm-eval) benchmarks a model against standardized public tasks such as MMLU. It won't tell you whether your retrieval pipeline returned the right passage, because your application isn't a concept it has. See the framework-versus-harness section above for the split.

Are these frameworks free to use?

License-wise, yes. DeepEval, Ragas, Opik, Evidently and Giskard are Apache-2.0; promptfoo, Inspect AI and lm-evaluation-harness are MIT. Both licenses permit commercial use, modification and redistribution. Arize Phoenix is the exception: Elastic License 2.0 permits self-hosting but not offering the software to third parties as a managed service. Judge-model API usage is billed separately by your provider.