Scraping 241 UK council planning portals – 2.6M decisions so far

Re:Scraping 241 UK council planning portals – 2.6M decisions... by mebkorea on Tuesday April 28, 2026 @03:49AM (#47931568)

Hadn't thought of that tbh. Worth a go on Liverpool especially... that's the AWS WAF one I'm currently blocked on and it is doing my head in. The challenge there is volume rather than access (~80k decisions to backfill), so even if an LLM gets through the wall I'd still need to script around it. But could be a way in for the initial cookie. Cheers for the tip and will look into it.

[ Parent | on HN ]

Re:Scraping 241 UK council planning portals – 2.6M decisions... by mobilio on Tuesday April 28, 2026 @07:05AM (#47932813)

can you share link to Liverpool so i can try it too please?

[ Parent | on HN ]

Re:Scraping 241 UK council planning portals – 2.6M decisions... by mebkorea on Tuesday April 28, 2026 @09:05AM (#47934020)

Sure, here you go: https://lar.liverpool.gov.uk/planning/index.html?fa=search

Heads up tho... it's behind AWS WAF with a JS challenge. Solving the challenge once works fine, but the WAF rate-limits the IP after ~10 requests and blocks for the rest of the day. So getting a session is doable, getting through 80,000 decisions is the hard bit. If you crack it I want to know! Cheers.

[ Parent | on HN ]