GetMentionedByAICloudflare Pay Per Crawl: what publishers can—and cannot—do
Cloudflare Pay Per Crawl uses HTTP 402 and authenticated crawler requests to let participating publishers charge for access. It remains a beta, not a universal web toll.
Cloudflare Pay Per Crawl is an experiment that lets participating site owners set a price for authenticated AI crawler requests. A crawler either presents payment intent and receives the content, or gets an HTTP 402 Payment Required response with pricing information. Cloudflare handles the infrastructure and acts as merchant of record.
Important limit: Pay Per Crawl is still a closed or private beta. Adding a 402 response by itself does not make every AI crawler pay.
How Pay Per Crawl works
- The publisher configures a zone price or dynamic pricing policy in Cloudflare.
- An authenticated crawler requests a protected URL with payment intent.
- If the offer satisfies the configured price, the request can receive a successful response and a charge record.
- If no acceptable payment intent is present, Cloudflare returns HTTP 402 with the required price.
- Cloudflare aggregates crawler charges and distributes publisher earnings under the beta program.
Cloudflare’s design uses Web Bot Auth so payment intent is tied to a verifiable crawler identity rather than a spoofable User-Agent string. That authentication layer is essential: a server cannot safely charge “GPTBot” merely because a request header says GPTBot.
Allow, block, charge, and negotiate are different policies
| Action | What happens | Best fit |
|---|---|---|
| Allow | Crawler receives content normally | Search or agent traffic you want |
| Block | Crawler is denied, commonly with 403 | Traffic you do not want to serve |
| Charge | Participating crawler must accept configured payment | Beta monetization tests |
| Custom 402 | Crawler receives a licensing or contact message | Negotiated access outside automatic payment |
Cloudflare’s broader AI Crawl Control is not identical to Pay Per Crawl. Paid Cloudflare customers can configure crawler actions and custom 402 messages more broadly, while automatic per-request settlement remains tied to the beta and participating crawlers.
Will charging crawlers hurt AI search visibility?
It can, depending on which URLs and crawlers are charged or blocked. Search crawlers need access to retrieve and index content for citation. A paywall that denies a search crawler removes that route to visibility. Training crawlers, search crawlers, and user-triggered agents create different value, so a single domain-wide policy is blunt.
- Keep discovery pages free if they help crawlers understand what content exists.
- Do not charge login, account, search, or other functional endpoints that contain no licensable content.
- Use URI-specific policies when premium archives have different economics from product and marketing pages.
- Check which crawlers currently reach the site before changing access with the AI crawler checker.
- Measure referral and conversion value separately from raw crawl volume.
Dynamic pricing and free paths
Cloudflare documents URI-pattern exclusions and dynamic pricing through a `crawler-price` response header. A Worker or origin can vary the price by content or request context. That makes a mixed policy possible: free homepage and category discovery, paid premium articles, and bypassed utility endpoints.
Should a publisher use it now?
Large publishers already using Cloudflare and carrying expensive archives have the clearest reason to test the beta. A small SaaS site whose main goal is being recommended by ChatGPT or Claude should first separate search visibility from training policy. Blocking or charging every AI request can cost more in discovery than it earns in crawl fees.
Primary sources: Cloudflare’s Pay Per Crawl documentation and launch explanation.
Common questions
Is Cloudflare Pay Per Crawl generally available?
No. Cloudflare describes automatic Pay Per Crawl as a closed or private beta.
Does HTTP 402 force an AI crawler to pay?
No. It communicates that payment or licensing is required. Automatic charging needs participating infrastructure and authenticated payment intent.
Can publishers charge only some URLs?
Cloudflare documents URI-pattern exclusions and dynamic pricing, allowing policies to vary across a site.