Sitemap and robots

The sitemap.xml and robots.txt Blume writes for crawlers, and the Content-Signal line that tells AI crawlers how they may use your docs.

Search crawlers and AI crawlers both start from the same two files at your site root. Blume writes both on every build, and the robots.txt carries the content-usage signals that AI crawlers look for, so this is where the search side and the agent side of discoverability meet.

Sitemap

Blume writes a sitemap.xml of every indexable page at build time. It needs an absolute deployment.site and lists every page except drafts, hidden, and noindex pages. On a versioned site, archived pages whose canonical points at their live equivalent are left out too — the live page is the one to index. On by default:

seo: {
  sitemap: true,
}

Ship your own public/sitemap.xml to take over — Blume never overwrites a file you place in public/.

Robots

Blume writes a robots.txt that allows all crawlers, declares your content signals, and adds a Sitemap: line pointing to the sitemap when one is available. On by default:

seo: {
  robots: true,
}

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

Sitemap: https://docs.example.com/sitemap.xml

Content signals

The Content-Signal line — the emerging content-usage convention — declares how AI crawlers may reuse your docs. Blume emits it on by default with every signal set to yes, matching its stance that docs are open to humans and agents alike:

• search — traditional and AI search indexing
• aiInput — grounding / RAG at answer time
• aiTrain — model training

Restrict any signal by setting it to false; the ones you leave out stay yes:

seo: {
  contentSignals: {
    aiTrain: false, // opt out of training, keep search + grounding
  },
}

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Set contentSignals: false to drop the declaration entirely:

seo: {
  contentSignals: false,
}

Content signals express a preference, not access control: they tell well-behaved crawlers how you’d like your content used, and it’s on the crawler to honor them. The same policy is mirrored as contentUsage in the agent readability manifest, so an agent that never reads robots.txt still sees it.

Ship your own public/robots.txt to take over.