---
title: Evals
description: blume eval gives your docs a test suite — an AI agent answers your users' questions using only the documentation, a judge grades the answers, and CI fails when the docs can't answer.
---

`blume audit` tells you whether crawlers can find your docs. `blume eval` tells you whether anyone can actually _use_ them: an AI agent reads your documentation the way a stranger would and tries to answer real user questions from it. When the docs don't state the answer, the run fails and names the page that should.

```bash
blume eval
```

```
blume eval  4 question(s) · Codex

  ✔ install-node-version         pass  1.00  14.2s
  ✔ custom-domain                pass  0.92  21.3s
  ✖ deploy-vercel                fail  0.40  38.9s
      missing: deployment: vercel() from blume/deploy
  ⊘ search-providers             skipped

  fix: content/docs/deployment.mdx  Docs could not answer: "How do I deploy to Vercel?" — missing: deployment: vercel() from blume/deploy

  2 passed · 1 failed · 1 skipped · 1m 42s
```

## How it works

Each question runs through two agent sessions, using an agent CLI you already have installed — [Codex](https://developers.openai.com/codex/cli) by default, or [Claude Code](https://claude.com/claude-code) with `--agent claude`. Blume holds no API keys and calls no model itself. With Codex, both sessions also run without Codex's shell, command, and image tools, and inherit none of your environment variables. With Claude Code, both sessions run with every built-in tool turned off and skip your Claude Code settings files and `CLAUDE.md`, so your hooks, plugins, and memory can't tell the reader what the docs don't. Your Claude Code login still works; set anything else it needs, such as `ANTHROPIC_API_KEY`, in your shell rather than in `settings.json`.

1. **The reader** answers the question using _only_ your documentation. It runs in an empty directory with its file, shell, and web tools disabled, connected to a private [MCP server](/docs/discoverability/mcp) that serves your docs — the same `search_docs`/`get_page` tools a real agent uses against your deployed site. It cannot read your repo, so it experiences the docs exactly like a fresh user: what isn't written doesn't exist.
2. **The judge** grades the answer against the facts you listed, with no tools at all. Paraphrase passes; a missing or contradicted fact fails — and so does "the documentation doesn't say."

The MCP snapshot is built from your content sources directly, so there is no need to run `blume build` first, and nothing is deployed or uploaded anywhere.

An answer the docs _can't_ support fails even when the agent's prior knowledge happens to be right — that's the point. Your docs are the only source that ships.

## Writing evals

Questions live in `evals.yaml` at the project root. To have an agent draft a starter file from your existing docs:

```bash
blume eval init
```

Or write it by hand:

```yaml
questions:
  - id: install-node-version
    question: What is the minimum Node.js version required?
    expected:
      - Node 22.19 or newer
    routes: /docs/quickstart
  - id: deploy-vercel
    question: How do I deploy to Vercel?
    expected:
      - run blume build
      - "server features need deployment: vercel() from blume/deploy"
    routes:
      - /docs/deployment
  - id: search-providers
    question: Which search providers are supported?
    expected:
      - Orama is the default, with no hosted service
    severity: warning # a miss warns instead of failing CI
    skip: true # temporarily excluded, reported as skipped
```

- `expected` lists the facts a correct answer must state, in substance — the judge accepts paraphrase and rejects contradiction.
- `routes` names the page(s) that should answer the question. A failure is then anchored to that page's source file in the report; a hint that no longer matches a page is warned about rather than silently dropped.
- `severity: warning` keeps a question in the report without failing CI; `skip: true` sits a question out entirely.

Write questions your users actually ask — the ones from support threads, GitHub issues, and onboarding calls. The best evals encode a promise your docs make ("zero-config deploys") as a question that breaks when a PR breaks the promise.

## Failing CI

The exit code is the contract: any failed question exits non-zero, except a `severity: warning` one — its miss is reported as a warning and never counts against the gate or `--threshold`. When the agent run itself fails — the reader or judge errors out rather than grading an answer — the report says `run failed:` and points at the question in your evals file instead of naming a docs page to fix, since the docs weren't graded. `--threshold` relaxes the gate to a passing fraction when you're digging out of a backlog:

```bash
blume eval                    # every question must pass
blume eval --threshold 0.8    # at least 80% must pass
blume eval --json             # machine-readable report on stdout
```

The JSON report carries the same `diagnostics` + `summary` shape as `blume validate --json` and `blume audit --json`, with the per-question results (answer, score, missing facts, and with Claude Code, cost) alongside.

Because each question is two model sessions, an eval run costs real money and minutes. With Claude Code, the per-question spend is printed as it runs and totaled in the summary; Codex doesn't report its spend, so a Codex run shows none. A sensible CI setup runs `blume eval` on docs changes rather than every push.

## Fixing the findings

Each failure names the missing facts and the page that should state them. To hand the whole report to the agent instead:

```bash
blume eval --fix
```

This writes the full JSON report to a file and opens the agent interactively with a prompt that walks it through each failed question: read the named page, add the missing facts in the page's voice, and rerun `blume eval` with the run's own `--agent`, `--file`, `--threshold`, and `--timeout` until everything passes. The session is interactive by design — you review the edits through the agent's own permission flow — and the agent is told never to delete questions or weaken expected facts to get to green.

## Flags

- `--agent codex|claude` — which agent CLI runs the reader and judge. Defaults to `codex`.
- `--file <path>` — the evals file, relative to the project root or absolute. Defaults to `evals.yaml`.
- `--threshold <0..1>` — minimum passing fraction before the run exits non-zero, with `severity: warning` misses counted as passing. Defaults to `1`. An empty value is an error, so `--threshold "$EVAL_THRESHOLD"` with the variable unset can't switch the gate off.
- `--timeout <seconds>` — reader time limit per question. Defaults to `180`, and can be at most `2147483` (about 24 days), the longest a timer can wait.
- `--json` — emit the report as JSON on stdout.
- `--fix` — after a failing run, hand the report to the agent to fix the docs interactively.
- `--verbose` — include the reader's full answer under each failure.
