# Shopify standard – tilpasset for Bokia.no # Sist oppdatert: 2026-09-09 # # Endringer 2026-09-09: # - Meta: meta-externalagent (AI-trening) stengt helt. # meta-webindexer (Meta AI-søk) tillatt på samme vilkår som OAI-SearchBot/PerplexityBot. # facebookexternalhit, meta-externalads og meta-externalfetcher er bevisst IKKE nevnt: # de følger *-gruppen (lenkeforhåndsvisninger, katalog- og annonsesjekk). # - MJ12bot og Pinterestbot: backend-sperrer lagt til. En crawler bruker KUN den mest # spesifikke gruppen som matcher, aldri * i tillegg – grupper med bare Crawl-delay var åpne overalt. # - GPTBot, anthropic-ai, Google-Extended: samme sperreliste som de andre AI-gruppene. # - FacebookBot er et utgått Meta-token; beholdt fordi det er harmløst. # Universell sitemap Sitemap: https://bokia.no/sitemap.xml Sitemap: https://bokia.no/sitemap_agentic_discovery.xml # 📍 Regler for alle crawlere (Googlebot, Bingbot osv.) User-agent: * # Backend / kasse / kunde Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /20642115/checkouts Disallow: /20642115/orders Disallow: /account Disallow: /a/downloads/-/* Disallow: /apple-app-site-association Disallow: /.well-known/shopify/monorail # Søk og anbefalinger (tynt innhold) Disallow: /search Disallow: /a/search Disallow: /recommendations/products Disallow: /*/recommendations/products # App-skript Disallow: /cdn/wpm/*.js # ❌ Filtrerte/sorterte URL-er Disallow: /collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /collections/*filter* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*ls=* # Crawl-budget rydding Disallow: /collections/mangler-bilde # 🤖 Google AdsBot (annonser) User-agent: AdsBot-Google Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /checkouts/ Disallow: /checkout Disallow: /account Disallow: /20642115/checkouts Disallow: /20642115/orders Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* # 🤖 SEO-analyseboter (lavere prioritet) User-agent: AhrefsBot Crawl-delay: 10 Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /account User-agent: AhrefsSiteAudit Crawl-delay: 10 Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /account User-agent: MJ12bot Crawl-delay: 10 Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /20642115/checkouts Disallow: /20642115/orders Disallow: /account Disallow: /search Disallow: /a/search Disallow: /recommendations/products Disallow: /*/recommendations/products Disallow: /cdn/wpm/*.js Disallow: /collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*filter* Disallow: /collections/mangler-bilde User-agent: Pinterestbot Crawl-delay: 1 Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /20642115/checkouts Disallow: /20642115/orders Disallow: /account Disallow: /search Disallow: /a/search Disallow: /recommendations/products Disallow: /*/recommendations/products Disallow: /cdn/wpm/*.js Disallow: /collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*filter* Disallow: /collections/mangler-bilde # Kjent uautorisert skraper User-agent: Nutch Disallow: / # ═══════════════════════════════════════════════════════════ # AI-søk / svar-crawlere: tillatt, men uten backend og tynt innhold # ═══════════════════════════════════════════════════════════ User-agent: ChatGPT-User Crawl-delay: 2 Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /20642115/checkouts Disallow: /20642115/orders Disallow: /account Disallow: /search Disallow: /a/search Disallow: /recommendations/products Disallow: /*/recommendations/products Disallow: /cdn/wpm/*.js Disallow: /collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*filter* Disallow: /collections/mangler-bilde User-agent: OAI-SearchBot Crawl-delay: 2 Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /20642115/checkouts Disallow: /20642115/orders Disallow: /account Disallow: /search Disallow: /a/search Disallow: /recommendations/products Disallow: /*/recommendations/products Disallow: /cdn/wpm/*.js Disallow: /collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*filter* Disallow: /collections/mangler-bilde User-agent: PerplexityBot Crawl-delay: 2 Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /20642115/checkouts Disallow: /20642115/orders Disallow: /account Disallow: /search Disallow: /a/search Disallow: /recommendations/products Disallow: /*/recommendations/products Disallow: /cdn/wpm/*.js Disallow: /collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*filter* Disallow: /collections/mangler-bilde User-agent: ClaudeBot Crawl-delay: 2 Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /20642115/checkouts Disallow: /20642115/orders Disallow: /account Disallow: /search Disallow: /a/search Disallow: /recommendations/products Disallow: /*/recommendations/products Disallow: /cdn/wpm/*.js Disallow: /collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*filter* Disallow: /collections/mangler-bilde User-agent: Applebot Crawl-delay: 2 Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /20642115/checkouts Disallow: /20642115/orders Disallow: /account Disallow: /search Disallow: /a/search Disallow: /recommendations/products Disallow: /*/recommendations/products Disallow: /cdn/wpm/*.js Disallow: /collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*filter* Disallow: /collections/mangler-bilde # Meta AI-søk (gir siteringer i Meta AI) – samme vilkår som OAI-SearchBot/PerplexityBot User-agent: meta-webindexer Crawl-delay: 2 Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /20642115/checkouts Disallow: /20642115/orders Disallow: /account Disallow: /search Disallow: /a/search Disallow: /recommendations/products Disallow: /*/recommendations/products Disallow: /cdn/wpm/*.js Disallow: /collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*filter* Disallow: /collections/mangler-bilde # ═══════════════════════════════════════════════════════════ # AI-trening med begrenset tilgang # ═══════════════════════════════════════════════════════════ User-agent: GPTBot Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /20642115/checkouts Disallow: /20642115/orders Disallow: /account Disallow: /search Disallow: /a/search Disallow: /recommendations/products Disallow: /*/recommendations/products Disallow: /cdn/wpm/*.js Disallow: /collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*filter* Disallow: /collections/mangler-bilde # anthropic-ai er et eldre token; ClaudeBot over er den aktive crawleren User-agent: anthropic-ai Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /20642115/checkouts Disallow: /20642115/orders Disallow: /account Disallow: /search Disallow: /a/search Disallow: /recommendations/products Disallow: /*/recommendations/products Disallow: /cdn/wpm/*.js Disallow: /collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*filter* Disallow: /collections/mangler-bilde # Google-Extended er et kontroll-token (Gemini-trening), ikke en egen crawler User-agent: Google-Extended Disallow: /admin Disallow: /cart Disallow: /carts Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /20642115/checkouts Disallow: /20642115/orders Disallow: /account Disallow: /search Disallow: /a/search Disallow: /recommendations/products Disallow: /*/recommendations/products Disallow: /cdn/wpm/*.js Disallow: /collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*filter* Disallow: /collections/mangler-bilde # ═══════════════════════════════════════════════════════════ # Helt stengt (AI-trening og skrapere) # ═══════════════════════════════════════════════════════════ # Meta AI-trening (respekterer robots.txt; regn med inntil 24 timer) User-agent: meta-externalagent Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Omgili Disallow: / # Utgått Meta-token, beholdt fordi det er harmløst User-agent: FacebookBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: /