DeepEval

What it does

DeepEval is an open-source Python framework for testing LLM applications such as AI agents, multi-turn chatbots, voice agents, RAG pipelines, and MCP systems. Test cases capture inputs, outputs, expected behavior, tools, context, and metadata, and metrics score them with explainable reasoning. Evaluations run locally in your own environment.

Teams write Pytest-style assertions for AI behavior and run them in CI/CD or as Python scripts with the deepeval test run command. The framework offers 50+ ready-to-use metrics, supports end-to-end, component-level, and trajectory-based evals, and can generate synthetic goldens and simulate conversations. Its CLI can also be driven by coding agents such as Claude Code, Codex, and Cursor.

It is aimed at AI engineers, data scientists, QA staff, and technical product managers. No account is needed to run it locally, though metrics that use an LLM judge need an LLM provider key such as OPENAI_API_KEY. Confident AI is a separate product for production monitoring and is not required.

What it can do

• Pytest-native unit testing of LLM outputs with deepeval test run
• 50+ research-backed metrics including hallucination, faithfulness, and answer relevancy
• Native conversational and voice evals, plus multi-modal text, image, and audio testing
• End-to-end, trajectory-based, and component-level evaluation of agents and RAG pipelines
• Synthetic golden and conversation simulation generation from knowledge bases or personas
• Custom metrics via plain-English criteria, G-Eval, DAG, and QAG techniques

Works with

• OpenTelemetry
• GitHub Actions
• GitLab CI
• Jenkins
• CircleCI
• Buildkite
• Azure Pipelines

Before you choose

• Metrics that use an LLM judge require an LLM provider key and incur that provider's model costs.
• Confident AI is a separate product for production monitoring; an account is only needed to send traces or results there.

Listed under

• Coding
• Code review and testing
• Agents
• AI agents
• Ci/cd
• Developer tools
• LLM evaluation
• Open source

Get started

Install DeepEval and follow the 5-minute quickstart; no account needed for local use.

1. Install the DeepEval Python package and create a first test case.
2. Set an LLM provider key such as OPENAI_API_KEY for metrics that use an LLM judge.
3. Run tests with deepeval test run and integrate into your CI/CD provider.

DeepEval is an open-source framework; verify any Confident AI plan pricing with the vendor. LLM provider costs for judge metrics are separate.

We couldn’t find published prices for DeepEval.

Sources for this overview

Product pages, documentation, and pricing checked 29 Oct 2026. Each source supports part of the overview; availability and prices can change.

• Introduction to DeepEval | DeepEval - The LLM Evaluation Framework ↗deepeval.com
• DeepEval - The LLM Evaluation Framework ↗deepeval.com

Company website

deepeval.com: product, pricing and policy pages

Hacker News

Launch posts and discussion

YC directory mirror

Batch and company name

Web search

Confirms the official site

Social accounts

Accounts the company links, and its posts on X

Human review

Checked by a person

manual_review

Last checked 29 Oct 2026. Every field keeps its source and date; unknown means we couldn’t confirm it.