Identify the crawler
Use published documentation to identify the user agent and its stated purpose before changing a control.
AI crawlers · technical reference
AI crawlers: identify the purpose before setting controls
Crawler controls depend on the crawler and purpose. Separate search discovery from training-related access rather than treating every AI crawler as interchangeable.
Direct answer
Keep the search question explicit.
Read the published crawler documentation and apply controls to the specific user agent and purpose. A control is not a guarantee of how any answer experience will use or cite content.
Reference guide
Concepts and evidence boundaries.
Separate discovery from training
OpenAI guidance distinguishes OAI-SearchBot for search discovery from GPTBot training controls; do not collapse those purposes.
Verify ordinary access
Robots directives are crawler access guidance. They do not promise crawling, inclusion, sources, or citations.
Primary sources
Official guidance reviewed 2026-08-16.
Publishers and developers FAQ
OpenAI. Platform guidance can change; recheck the source before acting.
OpenAI SearchBot
OpenAI. Platform guidance can change; recheck the source before acting.
Related guides
Continue with a focused question.
GEO guide
Return to the educational GEO reference hub.
Questions
What this reference does not promise.
Are all AI crawlers controlled the same way?
No. Controls and meanings depend on the documented crawler and purpose.
Have a real research question?
Inspect evidence in Gavix.
Use supported workflows and keep their boundaries visible.
Start free trial