PerplexityBot is Perplexity's crawler for surfacing and linking websites in its search results. Perplexity says it isn't used to crawl content for AI foundation models. A store that blocks it, in robots.txt or by an over-eager firewall, can't be linked as a source in Perplexity's answers.
PerplexityBot is Perplexity’s crawler for surfacing and linking websites in its search results. Perplexity says it isn’t used to crawl content for AI foundation models, so blocking PerplexityBot mainly stops your pages from being linked as sources in Perplexity’s answers.
What does PerplexityBot do?
It crawls pages so Perplexity search can cite and link them. Perplexity describes it as “designed to surface and link websites in search results on Perplexity”, and its user agent string ends in PerplexityBot/1.0 with a link to Perplexity’s bot page.
The second agent, Perplexity-User, works differently: it fetches a page when a person asks a question, and “generally ignores robots.txt rules” because a user requested it. So a robots.txt rule for the crawler doesn’t stop a shopper’s question from reaching your page, and that is usually what a store wants.
Should you allow the Perplexity crawler?
If you want to be a source in its answers, yes. Perplexity recommends “allowing PerplexityBot in your site’s robots.txt file and permitting requests from our published IP ranges”, and says changes can take up to 24 hours to show.
There’s no training trade-off to weigh here, unlike with GPTBot: Perplexity says this crawler isn’t used for AI foundation models. Remember the group rule from RFC 9309 and Google’s spec: if you write a group just for this crawler, it ignores your * rules, so repeat any disallow lines it should keep.
Why can a firewall block PerplexityBot by mistake?
Because bot protection runs before robots.txt is ever read. A web application firewall rule aimed at “AI bots” or unknown crawlers can turn the Perplexity crawler away while your robots.txt says it’s welcome, and nothing in the store admin shows it.
Defaults are moving too. Cloudflare’s AI bot settings sort bots into Search, Agent and Training, and from 15 September 2026 new domains block Training and Agent bots on pages that display ads while “Search will remain allowed”. A search crawler like this one sits in the allowed group, but an older blanket rule may not.
Perplexity’s fix is one allow rule with 2 conditions: the user agent contains PerplexityBot (or Perplexity-User) AND the source IP is in its published list. Its docs walk through that rule for Cloudflare and AWS WAF, and recommend the IP check because a user agent string alone can be copied by anyone.
How do you verify it’s really Perplexity?
Match the request’s IP address against perplexity.com/perplexitybot.json. On 17 October 2026 that file held 8 IPv4 ranges and was dated 7 February 2025; the Perplexity-User file held 4.
In our check of 30 jewelry, fashion and beauty stores on the same day, 1 of 25 readable robots.txt files named PerplexityBot, and none blocked it. Being crawlable doesn’t promise a citation; it only makes one possible. Whether Perplexity cites you then depends on whether your pages answer the question better than the alternatives. Our AI visibility audit checks crawler access and citations together, and our generative engine optimization work and the AI search optimization hub cover the rest.