# Default policy for all crawlers User-agent: * # NOTE: /admin is deliberately NOT disallowed. The login view already sends # , and blocking the path here # meant Google could never READ that directive - which is exactly how the admin # pages ended up under "Indexed, though blocked by robots.txt". Letting the # crawler in is what lets the noindex do its job and drop them from the index. # This is not a security boundary; AdminAuthFilter is. Disallow: /set_language # POST-only form handlers. Googlebot GETs them (it finds them as form actions) # and every hit is a guaranteed 404 - /register/send was showing up in the # Search Console "Not found" report. Nothing here is a page; the browser POSTs # are unaffected, robots.txt only speaks to crawlers. Disallow: /contact/send Disallow: /giftcard/send Disallow: /register/send Disallow: /api/landing-mail # Tracking query-param URLs. The clean URL is the canonical one; these # variants only add crawl noise and duplicate "crawled, not indexed" entries. # (The old ?lang= switcher URLs are NOT blocked: they 301 to the real language # page, and Google has to be able to crawl them to see that.) Disallow: /*?trk= Disallow: /*&trk= Disallow: /*?utm_ Disallow: /*&utm_ Disallow: /*?fbclid= Disallow: /*?gclid= # AI crawlers & assistants - explicitly allowed to read and cite the site. # (Each is listed so the "yes, you may crawl/cite" decision is intentional.) User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: CCBot Allow: / Sitemap: https://digischool.am/sitemap.xml