# govoridictionaries.com # /w/ (deep-link bouncer) and /promo/ (campaign landing page) stay crawlable on # purpose: they are the App Link / Universal Link targets, and Google's deep-link # validator fetches them — blocking them here is what makes Play and Analytics # report the links as misconfigured. They are kept out of the index by the # in their own markup, which only works # if the crawler is allowed to fetch the page and read it. User-agent: * Allow: / Disallow: /videos_full/ # Govori Web, the dictionaries at /sr-/ and /cnr-/ (web/robots.txt): its API, # audio clips and search page are not for crawling; its pages are. Disallow: /_/api/ Disallow: /_/audio/ Disallow: /sr-*/search Disallow: /cnr-*/search # Not for training Google's AI models (Gemini). Google-Extended is a robots.txt # token, not a crawler: Googlebot still crawls and indexes everything above. Set # here because Cloudflare's "block AI training" rule blocked Googlebot itself # (403 on the sitemap, 2026-10-05). User-agent: Google-Extended Disallow: / # Parallel's crawler for its AI-agent search API, which brings the site few # readers. On 2026-10-11 it was 91% of the dictionary Worker's requests (~3 a # second, each page rendered with its scripts and images): enough to spend the # Workers Free plan's 100,000 requests a day, which closed /sr-* until midnight # UTC. The account is on Workers Paid since 2026-10-11 (no daily cutoff; billed # per million requests), so these crawlers now cost money rather than uptime. # Shap-User, Parallel's agent acting for a person, is not a crawler and is not # named here. User-agent: ShapBot Disallow: / # The SEO tools' index crawlers (Ahrefs, Semrush): they serve those tools' users, # not the site's readers, and on 2026-10-11 they were the largest share after # ShapBot (~17,000 and ~6,500 requests a day). Their on-demand # audit bots (AhrefsSiteAudit, SiteAuditBot, SemrushBot-BA, -SI, …) have tokens # of their own and are not named here. User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / Sitemap: https://govoridictionaries.com/sitemap.xml Sitemap: https://govoridictionaries.com/_/sitemap.xml