The open internet is quietly installing a toll booth for machines. We measure the tolls — observed prices, paywalls, and block rates across the web — and publish them every week.
When an AI crawler requests a page, a growing number of sites answer with HTTP 402 Payment Required instead of the content. A few attach an actual number. In this scan, Stack Overflow quoted USD 0.50 per crawl to Anthropic's ClaudeBot through Cloudflare's pay-per-crawl, returned in the same x402 settlement format agents use to pay autonomously. That single quote is the leading edge of a market that barely has public prices. We exist to collect them.
Across the top 2,000 sites on the web, fewer than one in five block any given AI crawler in robots.txt. Among news and reference publishers, the rate is roughly four to five times higher — the sites whose text models most want are the sites fighting hardest to gate it. And publishers now discriminate between bots: training crawlers get blocked far more than the search bots that still send readers back.
These are the whole-web figures, refreshed weekly. The full per-domain table, publisher panel, and country cuts are in the data feed.
| AI crawler | Blocked in robots.txt | Rate |
|---|
No standard has won, so the crawl economy speaks in dialects. We fingerprint each one across the web and track its spread. Every dialect below was observed in the current scan.
Today's numbers can be copied. The history cannot. We snapshot the crawl economy on every scan, so subscribers can see how it moves — which bots are being blocked more, which countries are tightening, and whether the price of a crawl is climbing — week over week, month over month, and eventually year over year. A competitor who starts measuring later has no way to reconstruct the weeks in between.
Trend tracking began on this baseline scan. Month-over-month movements unlock in the Terminal tier as history accrues — see access →
→ New: World editions — AI-crawler blocking by country
Headline numbers are free and meant to be cited. Depth — full per-domain history, the publisher panel, per-country and per-language editions, and the machine-readable feed — is for subscribers. Fittingly, our own data is available to AI agents through pay-per-crawl: the index charges for crawls, in an index of what crawls cost.