Robots.txt File
Among other things, your robot.txt allows the various LLMs to search and train from content on your website. This is a very important file and should only be modified by an experienced website developer. The following is a list current tokens that should be listed in your robots.txt file, allowing the vatious AI search bots access to select folders. Note, these do change from time to time so please verifty with each AI before publishing.
# Default: block everything for all other crawlers
User-agent: *
Disallow: /
# ---- OpenAI ----
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Allow: /blog/
Allow: /docs/
Allow: /faq/
Disallow: /admin/
Disallow: /account/
Disallow: /checkout/
Disallow: /internal/
# ---- Google / Gemini ----
User-agent: Google-Extended
User-agent: Google-Bot
Allow: /blog/
Allow: /docs/
Allow: /faq/
Disallow: /admin/
Disallow: /account/
Disallow: /checkout/
Disallow: /internal/
# ---- Anthropic / Claude ----
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /blog/
Allow: /docs/
Allow: /faq/
Disallow: /admin/
Disallow: /account/
Disallow: /checkout/
Disallow: /internal/
# ---- Perplexity ----
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /blog/
Allow: /docs/
Allow: /faq/
Disallow: /admin/
Disallow: /account/
Disallow: /checkout/
Disallow: /internal/
# ---- Optional: broader AI training crawlers ----
# User-agent: Applebot-Extended
# User-agent: Amazonbot
# Allow: /blog/
# Allow: /docs/
# Allow: /faq/
# Disallow: /admin/
# Disallow: /account/
# Disallow: /checkout/
# Disallow: /internal/
Sitemap: https://www.yoursite.com/sitemap.xml