An AI crawler is an automated program run by an AI company that fetches web pages, either to train models, to index content for AI search answers, or to read a page a person asked about. The three types need different decisions, so one blanket block usually costs a store more than it saves.
An AI crawler is an automated program run by an AI company that fetches web pages to train models, index content for AI search answers or read a page a person asked about. Each type needs its own decision, so treating every AI crawler the same is the usual mistake.
What are the 3 types?
Training, search and user-triggered. The big AI companies now publish separate agents for each, and their docs say what each one is for: OpenAI, Anthropic and Perplexity.
The table names 9 agents from 4 operators. The ai search crawler type is the one a store usually wants to keep. Their published IP lists alone span 4 to 234 ranges each, so the AI bots hitting a store are many separate programs, not one. The user-triggered ones behave differently: OpenAI says robots.txt rules “may not apply” to ChatGPT-User, and Perplexity says its user agent “generally ignores robots.txt rules”.
Should you block AI crawlers?
Block by type, not as a group: an AI crawler that feeds training and one that feeds answers are different decisions. Blocking a training crawler opts your future content out of model training and leaves your AI search visibility alone; blocking a search crawler removes you from that assistant’s answers.
Most stores haven’t decided anything yet. In our check of 30 jewelry, fashion and beauty stores on 17 October 2026, 23 of 25 readable robots.txt files named no AI bots at all, and none blocked one site-wide. The 2 that did name them kept product pages open and only fenced off account, cart and checkout URLs, which is a sensible default for any store.
What does Cloudflare’s AI bot setting change?
It can block AI bots before robots.txt is ever read. Cloudflare sorts them into Search, Agent and Training presets, and its docs set new defaults for new domains from 15 September 2026: “bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed.”
Mixed-purpose crawlers that do both search and training are caught by any training block. If your store sits behind Cloudflare or another firewall, check those settings as well as robots.txt, because a firewall block never shows up in the robots.txt file you’d normally audit.
The same applies to Shopify, which says bot management at its network layer is handled for stores, while the robots.txt layer is yours to edit through the robots.txt.liquid template (Shopify Help Center).
How do you check which AI crawlers can reach your store?
Run 4 checks in order: robots.txt, firewall, rendered text, logs. The last one needs each operator’s published IP list; on 17 October 2026 those ranged from 4 ranges (Perplexity-User) to 234 (ChatGPT-User), and 3 had been regenerated in the previous 10 days, so fetch them fresh each time.
Rendered text matters because some AI crawlers don’t run JavaScript; Common Crawl says CCBot doesn’t, for example. A product description that only appears after a script runs is invisible to them, however good it is. Our AI visibility audit runs these checks with a citation test, our technical SEO work fixes what it finds, and the AI search optimization hub covers the rest.