> ## Documentation Index
> Fetch the complete documentation index at: https://agno-v2.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# What are Evals?

> Measure agent and team quality across expected answers, custom criteria, tool behavior, and performance.

Teams shipping agents need to know whether a prompt, model, tool, or context change improved the product. Evals turn expected behavior into repeatable checks that can run during development and in CI.

```python accuracy_eval.py theme={null}
from agno.agent import Agent
from agno.eval.accuracy import AccuracyEval
from agno.models.openai import OpenAIResponses
from agno.tools.calculator import CalculatorTools

agent = Agent(
    model=OpenAIResponses(id="gpt-5.4-mini"),
    tools=[CalculatorTools()],
)

evaluation = AccuracyEval(
    name="Calculator accuracy",
    model=OpenAIResponses(id="gpt-5.4-mini"),
    agent=agent,
    input="What is 10 factorial?",
    expected_output="3628800",
)

result = evaluation.run(print_results=True)
assert result is not None and result.avg_score >= 8
```

`AccuracyEval` runs the agent, then uses the evaluator model to compare its response with the expected output.

## Choose an Eval

| Question | Eval |
| - | - |
| Does the response match an expected answer? | [Accuracy](https://agno-v2.mintlify.site/evals/accuracy/overview.md) |
| Does the response meet product-specific quality criteria? | [Agent as Judge](https://agno-v2.mintlify.site/evals/agent-as-judge/overview.md) |
| Did the agent or team call the expected tools with the expected arguments? | [Reliability](https://agno-v2.mintlify.site/evals/reliability/overview.md) |
| How long does the code take and how much memory does it use? | [Performance](https://agno-v2.mintlify.site/evals/performance/overview.md) |
| Do many judge and reliability cases pass together? | [Eval Suites](https://agno-v2.mintlify.site/evals/suite/overview.md) |

Accuracy and agent-as-judge checks use a model to evaluate output. Reliability checks inspect recorded tool calls and arguments. Performance checks execute a function repeatedly and report runtime and memory statistics.

## Build a Regression Suite

Eval suites run multiple `Case` definitions against agents or teams. Each case can check custom criteria, expected tool calls, or both.

```python evals.py theme={null}
import sys

from agno.agent import Agent
from agno.eval import Case, cli
from agno.models.openai import OpenAIResponses
from agno.tools.calculator import CalculatorTools

agent = Agent(
    id="calculator-agent",
    model=OpenAIResponses(id="gpt-5.4-mini"),
    tools=[CalculatorTools()],
    instructions="Use calculator tools for arithmetic.",
)

CASES = (
    Case(
        name="factorial_uses_calculator",
        agent=agent,
        input="What is 10 factorial?",
        tags=("smoke",),
        criteria="States that 10 factorial equals 3628800.",
        expected_tool_calls=("factorial",),
    ),
)

if __name__ == "__main__":
    sys.exit(cli(CASES))
```

```bash theme={null}
python evals.py --tag smoke --json-output tmp/evals.json
```

The CLI returns a failing exit code when a selected case fails and can write a JSON report for CI artifacts. See [Eval Suites](https://agno-v2.mintlify.site/evals/suite/overview.md) for selectors, timeouts, judge modes, and programmatic execution.

## What to Evaluate

Start with behavior that matters to the product:

| Product requirement | Check |
| - | - |
| Support answers cite the correct policy | Accuracy against reviewed expected answers |
| Responses follow your tone and escalation rules | Agent-as-judge criteria |
| Refund requests call the approval tool | Reliability tool-call checks |
| Search stays within a latency budget | Performance thresholds tracked by your test harness |

Keep cases focused on one behavior so failures point to a clear prompt, model, context, or tool change.

## Next Steps

> **[Accuracy](https://agno-v2.mintlify.site/evals/accuracy/overview.md)**
>
> Compare responses with expected answers.

> **[Agent as Judge](https://agno-v2.mintlify.site/evals/agent-as-judge/overview.md)**
>
> Score custom quality criteria.

> **[Reliability](https://agno-v2.mintlify.site/evals/reliability/overview.md)**
>
> Check tool calls and arguments.

> **[Performance](https://agno-v2.mintlify.site/evals/performance/overview.md)**
>
> Measure runtime and memory usage.

> **[Eval Suites](https://agno-v2.mintlify.site/evals/suite/overview.md)**
>
> Run many judge and reliability cases in CI.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.