Evals

blume eval gives your docs a test suite — an AI agent answers your users' questions using only the documentation, a judge grades the answers, and CI fails when the docs can't answer.

blume audit tells you whether crawlers can find your docs. blume eval tells you whether anyone can actually use them: an AI agent reads your documentation the way a stranger would and tries to answer real user questions from it. When the docs don’t state the answer, the run fails and names the page that should.

blume eval

blume eval  3 question(s) · Claude Code

  ✔ install-node-version         pass  1.00  14.2s  $0.14
  ✖ deploy-vercel                fail  0.40  38.9s  $0.31
      missing: the adapter is auto-detected
  ⊘ search-providers             skipped

  fix: content/docs/deployment.mdx  Docs could not answer: "How do I deploy to Vercel?" — missing: the adapter is auto-detected

  2 passed · 1 failed · 1 skipped · 1m 42s · $0.45

How it works

Each question runs through two agent sessions, using an agent CLI you already have installed — Claude Code by default, or Codex with --agent codex. Blume holds no API keys and calls no model itself.

1. The reader answers the question using only your documentation. It runs in an empty directory with its file, shell, and web tools disabled, connected to a private MCP server that serves your docs — the same search_docs/get_page tools a real agent uses against your deployed site. It cannot read your repo, so it experiences the docs exactly like a fresh user: what isn’t written doesn’t exist.
2. The judge grades the answer against the facts you listed, with no tools at all. Paraphrase passes; a missing or contradicted fact fails — and so does “the documentation doesn’t say.”

The MCP snapshot is built from your content sources directly, so there is no need to run blume build first, and nothing is deployed or uploaded anywhere.

An answer the docs can’t support fails even when the agent’s prior knowledge happens to be right — that’s the point. Your docs are the only source that ships.

Writing evals

Questions live in evals.yaml at the project root. To have an agent draft a starter file from your existing docs:

blume eval init

Or write it by hand:

questions:
  - id: install-node-version
    question: What is the minimum Node.js version required?
    expected:
      - Node 22.12 or newer
    routes: /docs/quickstart
  - id: deploy-vercel
    question: How do I deploy to Vercel?
    expected:
      - run blume build
      - the output directory is dist
    routes:
      - /docs/deployment
  - id: search-providers
    question: Which search providers are supported?
    expected:
      - pagefind is the default
    severity: warning # a miss warns instead of failing CI
    skip: true # temporarily excluded, reported as skipped

• expected lists the facts a correct answer must state, in substance — the judge accepts paraphrase and rejects contradiction.
• routes names the page(s) that should answer the question. A failure is then anchored to that page’s source file in the report; a hint that no longer matches a page is warned about rather than silently dropped.
• severity: warning keeps a question in the report without failing CI; skip: true sits a question out entirely.

Write questions your users actually ask — the ones from support threads, GitHub issues, and onboarding calls. The best evals encode a promise your docs make (“zero-config deploys”) as a question that breaks when a PR breaks the promise.

Failing CI

The exit code is the contract: any failed question exits non-zero. --threshold relaxes the gate to a passing fraction when you’re digging out of a backlog:

blume eval                    # every question must pass
blume eval --threshold 0.8    # at least 80% must pass
blume eval --json             # machine-readable report on stdout

The JSON report carries the same diagnostics + summary shape as blume validate --json and blume audit --json, with the per-question results (answer, score, missing facts, cost) alongside.

Because each question is two model sessions, an eval run costs real money and minutes — the per-question spend is printed as it runs. A sensible CI setup runs blume eval on docs changes rather than every push.

Fixing the findings

Each failure names the missing facts and the page that should state them. To hand the whole report to the agent instead:

blume eval --fix

This writes the full JSON report to a file and opens the agent interactively with a prompt that walks it through each failed question: read the named page, add the missing facts in the page’s voice, and rerun blume eval until everything passes. The session is interactive by design — you review the edits through the agent’s own permission flow — and the agent is told never to delete questions or weaken expected facts to get to green.

Flags

• --agent claude|codex — which agent CLI runs the reader and judge. Defaults to claude.
• --file <path> — the evals file. Defaults to evals.yaml.
• --threshold <0..1> — minimum passing fraction before the run exits non-zero. Defaults to 1.
• --timeout <seconds> — reader time limit per question. Defaults to 180.
• --json — emit the report as JSON on stdout.
• --fix — after a failing run, hand the report to the agent to fix the docs interactively.
• --verbose — include the reader’s full answer under each failure.