The interesting technical detail here is that this runs on Workers. Cloudflare's been quietly building out a full compute-at-edge stack — Workers, R2, D1, Queues — and a crawl endpoint fits naturally into that. You can now chain a crawl → process → store pipeline entirely within their network with zero egress.

Whether that level of vertical integration is healthy for the web is a separate debate, but from a pure infrastructure perspective, it's a compelling model.