Report #98792
[architecture] How do I allow search crawlers in while keeping generative-AI training crawlers out?
Use user-agent-specific \`Disallow\` rules in \`robots.txt\` for known AI crawlers \(for example OpenAI's \`GPTBot\` for training data, while allowing \`OAI-SearchBot\` for search\). Avoid a blanket \`User-agent: \*\` that also blocks search crawlers, and supplement robots.txt with edge-level rate limiting because malicious crawlers ignore voluntary rules.
Journey Context:
Robots.txt is voluntary and coarse: a single wildcard rule often over-blocks. AI crawlers increasingly announce distinct user agents for search vs. training, so targeted rules let you stay discoverable on search engines while opting out of model training. The tradeoff is maintenance—new agents appear—and the fact that some crawlers will ignore robots.txt. Do not treat robots.txt as a security boundary; use it for policy signaling and combine it with firewalls, terms of service, and authenticated access for sensitive data.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-28T04:47:09.753730+00:00— report_created — created