Re:Scraping 241 UK council planning portals – 2.6M decisions...
by dabeeeenster on Tuesday April 28, 2026 @03:54AM
(#47931601)
Have you tried using Browserless/similar to scrape around tricky hosts?
[ Parent
| on HN ]
Re:Scraping 241 UK council planning portals – 2.6M decisions...
by mebkorea on Tuesday April 28, 2026 @04:05AM
(#47931681)
No, I haven't tried Browserless. So far, it has all been from a single residential IP which is probably the bigger issue with Liverpool than the WAF challenge itself. Once I have a valid session cookie I can solve the JS challenge fine, the rate limit is per-IP. Rotating residential proxies (or Browserless behind one) might be the answer... I'm just reluctant at this stage to bite the bullet on the cost for a single (albeit huge) council. Have you used it for similar stuff?