AI Crawler Access and Bot Governance

AI crawlers now account for a fast-growing share of requests to almost every Pakistani website, yet most SMEs have never made a deliberate decision about which ones to let in. WeProms Digital configures access policies through robots.txt and edge rules so the crawlers you want — Googlebot, PerplexityBot, and the answer crawlers behind ChatGPT and Claude — reach your content, and the ones you do not value are managed on purpose. For mobile-first, cash-on-delivery storefronts running on Cloudflare, getting this wrong can quietly choke the organic search traffic that pays the bills.

This is the infrastructure layer underneath AI discoverability. Where our generative engine optimization work focuses on what your content says so AI systems cite it, access control decides whether those systems can reach your content at all, and on what terms.

Two years ago, most agencies treated AI crawlers as a single category and flipped one toggle. That toggle no longer reflects how the web works. Major AI platforms have split their crawlers by purpose: one bot trains the model, a different bot fetches pages to answer a live user question, and a third may act on a user’s behalf. OpenAI runs GPTBot for training and OAI-SearchBot for answers. Anthropic runs ClaudeBot for training and a separate answer crawler for citations. Google separates Googlebot from Google-Extended. Apple separates Applebot from Applebot-Extended.

That split changes the decision entirely. You can decline to let your content train a competitor’s foundation model while still appearing in the AI answers that real prospects read. You can welcome the answer crawlers that drive citations and block the training crawlers that only consume bandwidth. But you can only make those choices deliberately if each crawler is named and routed on its own line — and most inherited robots.txt files still treat AI as one blob, or were last edited before the split happened.

For Pakistani SMEs the stakes are concrete. Bandwidth and origin costs matter when margins are thin; content scraping by training crawlers can compete with your own positioning; and a single misconfigured edge rule can drop your Google traffic overnight. A deliberate access policy turns a vague worry about AI into a written, reviewable stance.

The engagement starts with a full audit of the access surface, not just the robots.txt file. We read the current robots.txt, check whether a valid llms.txt exists at the root, review every Cloudflare and WAF rule that touches bot or AI traffic, and pull recent crawl logs to see who is actually arriving. The gap between what the files say should happen and what the logs show is usually where the real problems live.

From that baseline we build a per-crawler policy. robots.txt is rewritten so each significant user agent — GPTBot, OAI-SearchBot, ClaudeBot, the Claude answer crawler, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, and the core search bots — has an explicit allow or disallow rather than falling through to a vague default. We author or repair llms.txt with curated markdown links and descriptions pointing at the ten to twenty pages that most deserve AI attention, then validate it so it is not flagged as an incomplete manifest.

On the edge we tune Cloudflare and WAF rules so the policy actually holds. We place explicit allow rules for verified search crawlers above any generic bot or AI blocks, scope training blocks narrowly instead of site-wide, and confirm that llms.txt itself is reachable to the crawlers that should read it. Everything gets documented and version-controlled so the next change is a small commit, not a forensic investigation.

The single most expensive mistake in this space is not a robots.txt typo. It is an edge-layer block that quietly stops Googlebot from crawling while every server-side log looks healthy.

Here is how it happens. Cloudflare now classifies crawlers by behavior into search, training, and agent buckets, and it enforces those classifications at the WAF layer before a request ever reaches your origin or reads your robots.txt. Googlebot is treated as a mixed-use crawler because it serves both core search indexing and some AI purposes. When a broad training block is enabled — often a single toggle flipped during an unrelated task — the edge applies the most restrictive rule to mixed-use crawlers, and Googlebot can be denied on large portions of the site.

Because the block happens at the edge, your origin logs show nothing unusual. The traffic never arrives. The symptom is a slow, confusing drop in indexed pages and organic visits weeks later, with no error in Search Console that points clearly at the cause. The fix is structural: explicit allow rules for verified search crawlers placed above generic blocks, narrow scoping of any training block, and verification through crawl logs and live fetch tests before we call the work done. This is the check that separates a real access governance engagement from a checkbox robots.txt edit.

A policy is only useful if it is informed by measurement, and that means looking at what each crawler actually returns. We pair crawl log analysis with AI visibility audits. Logs tell us which crawlers hit the site, how often, and how much they cost you in bandwidth and origin load. Citation checks tell us whether those crawlers are translating into something real — your brand appearing in AI answers, your pages referenced by Perplexity or ChatGPT, your products surfacing in generative results.

That evidence drives a per-crawler verdict. A training crawler that hits the site hard and never produces a citation becomes a block candidate. An answer crawler that consistently drives visibility gets its access protected and its path to your best content smoothed. Some crawlers fall in between and earn a conditional rule. The point is to replace inherited defaults and gut feelings with a written stance you can defend and revisit as new crawlers appear — and new ones appear often.

Access governance does not replace your other technical SEO work; it protects it. A clean crawl foundation, structured data, and strong on-page SEO only pay off if the crawlers that matter can actually reach them, and an accidental edge block can negate months of that effort in a single setting change. Layered underneath our generative engine optimization work, deliberate access control checks whether the intended search crawlers can reach your important public pages; an optional text index is documentation, not an access or ranking control.

We document the policy, version-control the files, and schedule periodic re-checks because the crawler landscape keeps shifting. New user agents arrive, platforms reclassify behaviors, and edge defaults change without warning. A governance retainer keeps your access posture current instead of letting it drift back into the dangerous default state most sites quietly occupy.

This service is for Pakistani SMEs and growing brands whose websites matter to revenue and who have realized that AI traffic is now a real, measurable part of their web footprint. It fits ecommerce stores on Shopify and WooCommerce, service businesses competing on local and national search, SaaS and B2B firms whose content should be cited by AI assistants, and any team on Cloudflare that has ever enabled an AI or bot setting without checking the side effects. If you want to be visible in AI answers without giving away your content to every training crawler, and you want certainty that Googlebot keeps crawling the pages that pay the bills, this is the engagement that puts it in writing.