News for hackers. Stuff that matters.
Re:Scraping 241 UK council planning portals – 2.6M decisions... by codeulike on Tuesday April 28, 2026 @04:41AM (#47931911)
I'd be careful because even though its 'public' data, scraping it might not be legal due to TOS of the various sites.

I did a search for my postcode and got given results for a different area and council miles away

Re:Scraping 241 UK council planning portals – 2.6M decisions... by mebkorea on Tuesday April 28, 2026 @06:44AM (#47932644)
All fixed. The bug was in how I was building the postcode-to-council mapping.

The script was extracting prefixes from individual application addresses, and Ceredigion's database happened to have a chunk of records with HR* postcodes (data entry errors at source. The addresses are Welsh places like Aberystwyth and Borth, but with Hereford postcodes attached). Those errors polluted the lookup table, so HR* ended up pointing at Ceredigion.

Fixed by trusting only the manual postcode-to-council mappings for councils that have them, rather than supplementing from address extraction. The postcode you sent now correctly shows "not yet in coverage" rather than results from rural Wales. Cheers for flagging. That was a zinger of a bug.

Re:Scraping 241 UK council planning portals – 2.6M decisions... by mebkorea on Tuesday April 28, 2026 @04:50AM (#47931970)
Thanks for the feedback. On TOS: the same answer as I gave others... the data is statutorily public, I respect rate limits. That being said, I admit it's a grey area I haven't 100% nailed down. The postcode bug is more concerning. That shouldn't happen. Do you mind sharing which postcode or city/county? It could be that it's falling back to the wrong council because I don't have data for the right one, or it's a bug in my mapping. Either way, it needs fixing asap! Cheers for flagging.
Re:Scraping 241 UK council planning portals – 2.6M decisions... by codeulike on Tuesday April 28, 2026 @05:21AM (#47932165)
OK I have emailed your hello@planninglens... address with screenshots
Re:Scraping 241 UK council planning portals – 2.6M decisions... by mebkorea on Tuesday April 28, 2026 @05:28AM (#47932218)
Thanks. Will look into that right away!