By Hamza Ali · WeProms Digital · November 9, 2026
TL;DR: Blocking every AI crawler is one robots.txt line — and for most Dubai restaurants it is the wrong line. Your pages already sit in indexes sold to Perplexity and Cohere; the block only removes your menu from answers. Allow the named crawlers, keep the menu readable, and check monthly what the assistants say.
robots.txt — the small text file every website publishes to tell crawlers which pages they may read — has become an emotional document in 2026. Owners of Pakistani and halal restaurants across Dubai read that AI companies copy menus without asking, and respond by disallowing every bot with AI in its name. The instinct is understandable; the mechanics do not cooperate. A 40-seat karahi house in Al Karama turning over AED 4,200 on a weekend family booking (illustrative figure — substitute your own till average) has more to lose from invisibility than from being read. This teardown examines the block-everything strategy as it actually behaves for a Dubai restaurant today.
What this gets right
The grievance is real. On 2 November 2026 the New York Times sued Brave, alleging the company copied more than 30 billion webpages using a crawler that never identified itself, then sold access through its Search API to AI firms — Brave’s own API page names Cohere, Mistral, Perplexity and You.com as users, and the complaint names five AI buyers, Search Engine Journal reports. A Dubai restaurant owner who feels copied is not paranoid. The copy happened.
The control is also real, and free. Cloudflare ships AI Crawl Control on every plan, including the free tier, letting a site allow or block each named AI crawler individually and watch the visits in a dashboard. A restaurant on Cloudflare can flip GPTBot off in two clicks.
And training-use opt-outs are maturing: on 30 October 2026, Google, Microsoft and OpenAI announced they will honour a shared robots.txt AI-training control, with Google treating it like its existing Google-Extended setting. As a rights statement, the block-everything stance is coherent. As a growth decision, it misfires — for three reasons that follow.
Where this breaks
Blocking named crawlers feels decisive. It is not. The first break: you cannot block a crawler with no name. Brave’s help pages state its crawler “does not advertise a differentiated user agent” and follows Google’s rules instead — whatever Googlebot may read, Brave reads. A robots.txt line that keeps Brave out also keeps Google out, and escaping Brave’s index requires a noindex tag that removes the page from Google as well. For a Karama restaurant whose Google Business Profile drives the Friday crowd, that trade is unworkable.
The second break: the copy already happened. Type site: followed by your domain into search.brave.com — everything listed sits in the index Brave sells. The Times complaint quotes Brave describing one API product as handing over “extracted page chunks and source metadata in a shape and length that satisfies the needs of an LLM without need for further scraping.” Most owners miss the copy that already happened. Blocking today closes the door after the van has left.
An agent choosing where its user eats dinner might pull menus from 1,000 restaurant websites, and the user goes to one — the other 999 restaurants carry a cost with no benefit, as Cloudflare CEO Matthew Prince described on his Q3 2026 earnings call.
That is the third break, read carefully. Cloudflare’s position is that agent traffic is becoming “sort of the new version of SEO” — OpenAI already pays Cloudflare for hints when a page changes, and Prince used a restaurant updating its prices as his example of a page worth re-reading. When the assistant pulls those 1,000 menus, a site that blocked the named crawlers simply is not in the pile. Block every named crawler and the assistant answers with a rival’s menu. The 999-versus-1 framing, meanwhile, is a Cloudflare-scale argument: nonhuman traffic could reach 1,000 times human traffic within five years across the internet — an infrastructure number about the whole web, not a dinner-time query in Al Karama. It says nothing about your 40 seats.
One more break, and it is the quietest: many owners never chose to block at all. Cloudflare flipped the default for new sites in mid-2025, so zones created since then block AI crawlers out of the box until the owner changes the policy. A restaurant site built last year may be invisible to ChatGPT today with nobody having decided anything. Check the dashboard before assuming intent.
The hidden cost of the wrong block
The crawl-cost argument is a rounding error at restaurant scale. One assistant reading a menu page costs a fraction of a fils in bandwidth; the citation that same read can produce is worth a table. The asymmetry runs entirely one way for a single-location operator.
Here is the claim, stated plainly so it can be disagreed with: for a 40-seat karahi house, blocking GPTBot protects nothing the marketplaces have not already shipped. Talabat, Deliveroo and Uber Eats list your menu, your prices and your ratings on their domains today, and no robots.txt line touches those pages. The only pages the block governs are the ones you own — the pages that could carry your name into an answer instead of a marketplace’s. The travel sector has already measured what that is worth: Tripadvisor’s CEO told analysts AI traffic is under 1 percent of visits but converts at close to five times the average on Viator, and about twice on Tripadvisor. That is travel-booking data, not restaurant data — it proves nothing about iftar covers in Dubai. It does show the direction: when someone asks an assistant, they have usually already decided to buy.
Then there is freshness. If OpenAI pays to know when a restaurant’s prices change, a readable, current menu page is infrastructure, not decoration. A blocked or abandoned page means the assistant answers from last Ramadan’s prices — the same failure mode as a Kuwait mandi restaurant showing wrong Ramadan hours, except the correction never comes because nobody re-read a blocked page. Ramadan 1448 is expected to begin around 8 February 2027, subject to moon sighting, The National’s UAE calendar projection notes; iftar menus and hours published six weeks before that date are what the assistants will quote all month.
What overseas Pakistani restaurant owners should do instead
The curated stance beats both extremes. Allow GPTBot, PerplexityBot and ClaudeBot — the named answer-engine crawlers — because those reads feed the answers diners actually see. Block scrapers you can name and verify. Watch the new shared AI-training token as providers switch it on, since it separates “may answer from my page” from “may train on my page” for the first time. Cloudflare’s Pay Per Crawl meanwhile lets a site charge per read where the option is offered, which suits publishers more than restaurants but completes the picture.
| Approach | Appears in AI answers | Menu stays current | Training control | Main risk |
|---|---|---|---|---|
| Block everything | No | No — nobody re-reads | Total, in theory | Rivals answer instead; Brave-style crawlers pass anyway |
| Allow everything | Yes | Yes | None | Training use without consent |
| Curated access | Yes | Yes | Per-crawler, improving | Needs a 10-minute monthly check |
Allow GPTBot and PerplexityBot and your live hours can be quoted in the answer. That requires the menu and opening hours to live as text on your own domain — not only as an Instagram image, a PDF, or a Talabat listing — and to be updated the week prices change. The community analogy holds: the uncle everyone asks for restaurant recommendations was always an unpaid marketing channel; the assistant is that uncle now, with a longer memory. Hand it one line to say. A practical step for owners who want the setup handled: paste this to whoever manages the site — Please allow GPTBot, PerplexityBot and ClaudeBot in robots.txt and in Cloudflare AI Crawl Control, keep our menu and hours as text on our own domain, and confirm the iftar menu replaces the old one before 8 February 2027 (Ramadan start, subject to moon sighting).
Then check monthly. Run three questions past ChatGPT and Perplexity — best karahi in Al Karama, iftar buffet near me, late dinner Karama — and note who gets named, using the method for checking what ChatGPT says about your business. Run the site: check on search.brave.com in the same sitting. Ten minutes, once a month, is the entire maintenance load.
Read next: how to get ChatGPT to recommend your Dubai oud shop and what to do when AI answers quote your restaurant’s website and link someone else’s.
WeProms Digital runs AI crawler access and bot governance setups for restaurants and retailers — a robots.txt and Cloudflare policy audit, a menu-readability check, and a monthly citation report, built for overseas Pakistani and halal businesses across Dubai, the wider UAE and the Gulf. It pairs naturally with our GEO and AI discoverability programme for owners who want the citation work handled end to end. Message hello@weproms.com, WhatsApp +92 300 0133399, or the contact page — Gulf business hours.
Frequently Asked Questions
Will allowing AI crawlers slow my restaurant website down?
No, at restaurant scale. The 1,000-times-human projection describes traffic across the whole internet, not one domain’s bill; a menu page read by a handful of assistants costs fractions of a fils in bandwidth. Cloudflare also does not charge customers for bot traffic it blocks on their behalf. The slowdown risk is a big-publisher problem, not a Karama problem.
How do I check whether AI companies already have my pages?
Open search.brave.com, type site: followed by your domain, and read what appears. Everything listed sits in the index Brave sells through its Search API to companies including Perplexity, Cohere and Mistral. The check takes 30 seconds and no account, and it works the same for a Dubai domain as anywhere else.
If I allow GPTBot, is ChatGPT copying my recipes?
Crawling to answer is not the same as training. Google, Microsoft and OpenAI have agreed to honour a shared robots.txt AI-training control, and Google already treats opt-outs through its Google-Extended setting, which also covers grounding in Gemini Apps. Reading a menu page to answer a diner’s question leaves the recipe in your kitchen and the citation on your site.
Does this change my Talabat or Deliveroo listing?
No. Marketplace listings live on the platforms’ own domains under their crawl policies, and your robots.txt has no authority there. That is precisely why your own site matters: it is the only page where you decide what assistants may read, and where a citation sends the diner to you rather than to a marketplace reorder screen.
What does WeProms charge to sort this out?
Pricing depends on scope — a one-time robots.txt and Cloudflare policy audit costs less than the monthly review retainer that follows it. WeProms quotes in writing before any work starts, and the audit itself usually completes within a week. Email hello@weproms.com or message the WhatsApp line with your domain for a scoped quote.
Sources & References
- Search Engine Journal — Cloudflare CEO Likens AI Agent Optimization To A New Version Of SEO — November 8, 2026
- Search Engine Journal — Brave Sells Its Search Index To AI, And Your Website Is Probably In It — November 6, 2026
- Search Engine Journal — Tripadvisor Says SEO Will Still Weigh On Growth In 2027 — November 8, 2026
- Search Engine Journal — Google Rank Data Issues, Lighthouse Update & AI Opt-Out — November 7, 2026
- Cloudflare — AI Crawl Control documentation — accessed November 2026
- Cloudflare — AI Crawler and Agent access policies (bot solutions) — accessed November 2026
- Cloudflare Blog — Introducing Pay Per Crawl — July 2025
- Brave — Search API — accessed November 2026
- The National — Ramadan 2027 UAE dates and Eid Al Fitr holiday — October 6, 2026
Additional reading from industry feeds:
