robots.txt for AI crawlers: separate search, training, and user actions
How to configure AI crawler rules without confusing search visibility, model training controls, and user-triggered retrieval.
Direct answer
Configure each documented crawler according to its actual role. For OpenAI, OAI-SearchBot controls eligibility for ChatGPT search crawling, while GPTBot relates to potential model-training use; ChatGPT-User is used for user-triggered actions and may not follow robots.txt in the same way. Do not assume that one rule controls every AI product.
Map the business decision before writing directives
A site may want inclusion in search-style answers while declining training use, or may want to block automated crawling of a sensitive section while keeping public documentation discoverable. Those are separate choices and should be expressed with the documented user agent for each provider.
Start from the provider's current documentation, not a copied list from a blog post. User-agent names, roles, and IP ranges can change. Keep a review date beside your internal crawler policy.
Understand the OpenAI user-agent split
OpenAI documents OAI-SearchBot for surfacing sites in ChatGPT search, GPTBot for crawling content that may be used to improve foundation models, and ChatGPT-User for certain user-triggered actions. OpenAI states that these settings are independent.
That means allowing OAI-SearchBot while disallowing GPTBot can express a search-versus-training preference. It does not guarantee appearance in an answer, and robots.txt is not an authorization system for private content.
Keep private resources private at the application layer
robots.txt is public and advisory. Never use it to protect account pages, source archives, customer reports, staging environments, or sensitive API routes. Require authentication and authorization, avoid exposing secrets in rendered HTML, and return appropriate status codes.
Disallowed paths can still be known from links or the robots file itself. For pages that should not appear in search but may be fetched by users, use the appropriate noindex mechanism and prevent those URLs from entering the sitemap.
Test the effective policy
Fetch /robots.txt without cookies, verify the content type and status, and evaluate the most specific matching group for each crawler. Confirm that the sitemap is declared and that blocked dashboard routes do not appear in public navigation or the sitemap.
Implementation checklist
- ✓Every explicit AI user-agent rule maps to a documented business decision.
- ✓Search and training crawlers are not treated as interchangeable.
- ✓Private data is protected by authentication, not robots.txt.
- ✓Public canonical content remains reachable by allowed search crawlers.
- ✓Blocked routes are absent from the XML sitemap.
- ✓The policy is reviewed against current provider documentation.
Frequently asked questions
Can I allow ChatGPT search but block OpenAI training crawling?
OpenAI documents OAI-SearchBot and GPTBot as independent controls, so a site can express those separate preferences in robots.txt.
Will allowing an AI crawler guarantee citations?
No. Allowing crawl access only removes one possible barrier. Retrieval, ranking, answer generation, and citation decisions remain product-specific.
Can robots.txt secure a private dashboard?
No. Use application authentication and authorization. A robots.txt disallow rule does not prevent a person or non-compliant client from requesting the URL.
Primary sources
These references support the standards and product behavior described above. They do not imply endorsement of Index Instrument.