# robots.txt — averon-systems.com
# Reviewed 2026-08-18. Re-read this file end to end quarterly; new AI crawler
# tokens appear faster than SEO tooling updates its defaults.
#
# Deployed by CI from business/02-marketing/website/ via .github/workflows/
# deploy-sites.yml on every push that touches that directory. Do not edit this
# file in S3 — the next deploy overwrites it.
#
# POLICY: allow everything, everywhere.
#
# This is a deliberate decision, not a default. There is one page, all of it is
# public marketing copy, and there is nothing here worth protecting from a
# scraper. The asymmetry that matters:
#
#   - Blocking a retrieval crawler removes us from that engine's answers
#     entirely. That is a direct loss of a lead channel.
#   - Allowing a training crawler costs us nothing we care about, because the
#     "IP" on this page is a sales argument we want repeated.
#
# The common 2026 advice is "block training bots, allow retrieval bots." That
# advice is written for publishers who monetise pageviews. We monetise being
# recommended. Do not copy it here.
#
# REVISITED 2026-08-22 and DELIBERATELY UNCHANGED. Eric asked for ai-train=no
# during a site-wide pass; the policy above was re-read and it holds, so nothing
# was changed. The reasoning, restated because it will come up again: there is no
# proprietary content on this site — it is all sales copy whose entire job is to
# be repeated. A model trained on it that later recommends this business has done
# exactly what the copy was written for. Blocking training protects an asset that
# does not exist here, at the cost of an emerging referral channel.
#
# The related point people get backwards: llms.txt is NOT the mechanism for AI
# visibility. Ahrefs analysed 137,000 domains and found 28% publish one, 97% of
# those files received ZERO traffic, and 96% of the requests that did arrive were
# bots (ahrefs.com/blog/llmstxt-study). This site ships one because it is 6KB and
# harmless, not because it is a strategy. The real mechanisms are the 17
# schema.org types in index.html and clean, quotable answer-first copy.
#
# If this ever needs to flip, the correct change is Google-Extended and GPTBot to
# Disallow while LEAVING OAI-SearchBot, PerplexityBot and ChatGPT-User allowed —
# see the token note below. Flipping the wrong pair silently removes this site
# from AI answers while still allowing training, which is the opposite of intent.
#
# The single most-often-misconfigured line in this space: GPTBot is OpenAI's
# TRAINING crawler. OAI-SearchBot is what decides whether we can be cited in
# ChatGPT. Blocking GPTBot does not remove us from ChatGPT answers; blocking
# OAI-SearchBot does. Same company, different tokens, opposite consequences.
# Likewise Google-Extended controls Gemini TRAINING only and has no effect on
# Google Search indexing — a Disallow there does not "protect" rankings, it
# just opts us out of an AI surface.

# ---- Traditional search ----
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: Applebot
Allow: /

# ---- AI retrieval / citation (these earn referral traffic) ----
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Amazonbot
Allow: /

# ---- AI training (allowed on purpose — see policy note above) ----
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

User-agent: CCBot
Allow: /

User-agent: cohere-ai
Allow: /

# ---- Everyone else ----
User-agent: *
Allow: /

Sitemap: https://averon-systems.com/sitemap.xml