News for hackers. Stuff that matters.
Re:Scraping 241 UK council planning portals – 2.6M decisions... by mebkorea on Tuesday April 28, 2026 @04:25AM (#47931810)
Planning data on gov.uk is really patchy and not useful for what I want. There's planning.data.gov.uk which has some boundary/policy data but no actual decisions. The decisions only exist on council portals, which is the whole reason this project exists. On patent tribunal, I haven't looked into that one specifically but a few general gov.uk tips: most gov.uk content is actually clean HTML (way easier than council portals), so if requests isn't working it's usually either JS-rendered content (Playwright fixes this) or session/cookie weirdness. Things that have helped me elsewhere: Playwright with page.wait_for_selector rather than networkidle, copying real browser headers wholesale (not just User-Agent), and checking if there's a hidden JSON API behind the page (open devtools → Network tab → look for XHR/fetch requests when you click search). Often there's a clean JSON endpoint that the page is using, which is way easier to scrape than the rendered HTML.
Re:Scraping 241 UK council planning portals – 2.6M decisions... by pbhjpbhj on Wednesday April 29, 2026 @03:40PM (#47953385)
Thanks for your help, I'll have a tinker with Playwright and see if I can get what I want.