# govoridictionaries.com
# /w/ (deep-link bouncer) and /promo/ (campaign landing page) stay crawlable on
# purpose: they are the App Link / Universal Link targets, and Google's deep-link
# validator fetches them — blocking them here is what makes Play and Analytics
# report the links as misconfigured. They are kept out of the index by the
# in their own markup, which only works
# if the crawler is allowed to fetch the page and read it.
User-agent: *
Allow: /
Disallow: /videos_full/
# Govori Web, the dictionaries at /sr-/ and /cnr-/ (web/robots.txt): its API,
# audio clips and search page are not for crawling; its pages are.
Disallow: /_/api/
Disallow: /_/audio/
Disallow: /sr-*/search
Disallow: /cnr-*/search
# Not for training Google's AI models (Gemini). Google-Extended is a robots.txt
# token, not a crawler: Googlebot still crawls and indexes everything above. Set
# here because Cloudflare's "block AI training" rule blocked Googlebot itself
# (403 on the sitemap, 2026-10-05).
User-agent: Google-Extended
Disallow: /
# Parallel's crawler for its AI-agent search API, which brings the site few
# readers. On 2026-10-11 it was 91% of the dictionary Worker's requests (~3 a
# second, each page rendered with its scripts and images): enough to spend the
# Workers Free plan's 100,000 requests a day, which closed /sr-* until midnight
# UTC. The account is on Workers Paid since 2026-10-11 (no daily cutoff; billed
# per million requests), so these crawlers now cost money rather than uptime.
# Shap-User, Parallel's agent acting for a person, is not a crawler and is not
# named here.
User-agent: ShapBot
Disallow: /
# The SEO tools' index crawlers (Ahrefs, Semrush): they serve those tools' users,
# not the site's readers, and on 2026-10-11 they were the largest share after
# ShapBot (~17,000 and ~6,500 requests a day). Their on-demand
# audit bots (AhrefsSiteAudit, SiteAuditBot, SemrushBot-BA, -SI, …) have tokens
# of their own and are not named here.
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
Sitemap: https://govoridictionaries.com/sitemap.xml
Sitemap: https://govoridictionaries.com/_/sitemap.xml