robots.txt (AI Crawler Policy)

What It Is

robots.txt declares crawler access at the site root. For LLM search, the decisive insight is that AI user-agents fall into five functional categories, and you must treat them differently:

  1. Training crawlers (GPTBot, ClaudeBot) — block to stay out of training datasets.
  2. Search / retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) — block these and you won't be cited. Opt in if you want AI visibility.
  3. User-triggered fetchers (ChatGPT-User, Perplexity-User) — generally ignore robots.txt because they're real user requests.
  4. Opt-out tokens (Google-Extended, Applebot-Extended) — directives that control training; they never crawl and never appear in logs.
  5. Undeclared / masquerading scrapers — some ignore robots.txt entirely (per Cloudflare 2026-01).

How It Applies to Marketing Factory

A naive or missing robots.txt leaves AI-crawler behavior implementation-defined. The factory should ship a deliberate, explicit policy that opts in to the search/retrieval crawlers it wants citations from (and makes a conscious choice on training crawlers). This is part of the ~3-hour technical baseline that llm-search depends on — pair it with llms-txt, schema-markup, a sitemap, and IndexNow.

  • llm-search — crawler policy gates whether you can be cited at all
  • llms-txt — the companion machine-readable shortlist file
  • schema-markup — the rest of the extractability plumbing

Referenced from: llm-search-visibility-and-content-metrics