Evals

blume eval gives your docs a test suite — an AI agent answers your users' questions using only the documentation, a judge grades the answers, and CI fails when the docs can't answer.

blume audit tells you whether crawlers can find your docs. blume eval tells you whether anyone can actually use them: an AI agent reads your documentation the way a stranger would and tries to answer real user questions from it. When the docs don’t state the answer, the run fails and names the page that should.

blume eval

blume eval  4 question(s) · Codex

  ✔ install-node-version         pass  1.00  14.2s
  ✔ custom-domain                pass  0.92  21.3s
  ✖ deploy-vercel                fail  0.40  38.9s
      missing: deployment: vercel() from blume/deploy
  ⊘ search-providers             skipped

  fix: content/docs/deployment.mdx  Docs could not answer: "How do I deploy to Vercel?" — missing: deployment: vercel() from blume/deploy

  2 passed · 1 failed · 1 skipped · 1m 42s

How it works

Each question runs through two agent sessions, using an agent CLI you already have installed — Codex by default, or Claude Code with --agent claude. Blume holds no API keys and calls no model itself. With Codex, both sessions also run without Codex’s shell, command, and image tools, and inherit none of your environment variables. With Claude Code, both sessions run with every built-in tool turned off and skip your Claude Code settings files and CLAUDE.md, so your hooks, plugins, and memory can’t tell the reader what the docs don’t. Your Claude Code login still works; set anything else it needs, such as ANTHROPIC_API_KEY, in your shell rather than in settings.json.

1. The reader answers the question using only your documentation. It runs in an empty directory with its file, shell, and web tools disabled, connected to a private MCP server that serves your docs — the same search_docs/get_page tools a real agent uses against your deployed site. It cannot read your repo, so it experiences the docs exactly like a fresh user: what isn’t written doesn’t exist.
2. The judge grades the answer against the facts you listed, with no tools at all. Paraphrase passes; a missing or contradicted fact fails — and so does “the documentation doesn’t say.”

The MCP snapshot is built from your content sources directly, so there is no need to run blume build first, and nothing is deployed or uploaded anywhere.

An answer the docs can’t support fails even when the agent’s prior knowledge happens to be right — that’s the point. Your docs are the only source that ships.

Writing evals

Questions live in evals.yaml at the project root. To have an agent draft a starter file from your existing docs:

blume eval init

Or write it by hand:

questions:
  - id: install-node-version
    question: What is the minimum Node.js version required?
    expected:
      - Node 22.19 or newer
    routes: /docs/quickstart
  - id: deploy-vercel
    question: How do I deploy to Vercel?
    expected:
      - run blume build
      - "server features need deployment: vercel() from blume/deploy"
    routes:
      - /docs/deployment
  - id: search-providers
    question: Which search providers are supported?
    expected:
      - Orama is the default, with no hosted service
    severity: warning # a miss warns instead of failing CI
    skip: true # temporarily excluded, reported as skipped

• expected lists the facts a correct answer must state, in substance — the judge accepts paraphrase and rejects contradiction.
• routes names the page(s) that should answer the question. A failure is then anchored to that page’s source file in the report; a hint that no longer matches a page is warned about rather than silently dropped.
• severity: warning keeps a question in the report without failing CI; skip: true sits a question out entirely.

Write questions your users actually ask — the ones from support threads, GitHub issues, and onboarding calls. The best evals encode a promise your docs make (“zero-config deploys”) as a question that breaks when a PR breaks the promise.

Failing CI

The exit code is the contract: any failed question exits non-zero, except a severity: warning one — its miss is reported as a warning and never counts against the gate or --threshold. When the agent run itself fails — the reader or judge errors out rather than grading an answer — the report says run failed: and points at the question in your evals file instead of naming a docs page to fix, since the docs weren’t graded. --threshold relaxes the gate to a passing fraction when you’re digging out of a backlog:

blume eval                    # every question must pass
blume eval --threshold 0.8    # at least 80% must pass
blume eval --json             # machine-readable report on stdout

The JSON report carries the same diagnostics + summary shape as blume validate --json and blume audit --json, with the per-question results (answer, score, missing facts, and with Claude Code, cost) alongside.

Because each question is two model sessions, an eval run costs real money and minutes. With Claude Code, the per-question spend is printed as it runs and totaled in the summary; Codex doesn’t report its spend, so a Codex run shows none. A sensible CI setup runs blume eval on docs changes rather than every push.

Fixing the findings

Each failure names the missing facts and the page that should state them. To hand the whole report to the agent instead:

blume eval --fix

This writes the full JSON report to a file and opens the agent interactively with a prompt that walks it through each failed question: read the named page, add the missing facts in the page’s voice, and rerun blume eval with the run’s own --agent, --file, --threshold, and --timeout until everything passes. The session is interactive by design — you review the edits through the agent’s own permission flow — and the agent is told never to delete questions or weaken expected facts to get to green.

Flags

• --agent codex|claude — which agent CLI runs the reader and judge. Defaults to codex.
• --file <path> — the evals file, relative to the project root or absolute. Defaults to evals.yaml.
• --threshold <0..1> — minimum passing fraction before the run exits non-zero, with severity: warning misses counted as passing. Defaults to 1. An empty value is an error, so --threshold "$EVAL_THRESHOLD" with the variable unset can’t switch the gate off.
• --timeout <seconds> — reader time limit per question. Defaults to 180, and can be at most 2147483 (about 24 days), the longest a timer can wait.
• --json — emit the report as JSON on stdout.
• --fix — after a failing run, hand the report to the agent to fix the docs interactively.
• --verbose — include the reader’s full answer under each failure.