News for hackers. Stuff that matters.
Re:Scraping 241 UK council planning portals – 2.6M decisions... by dabeeeenster on Tuesday April 28, 2026 @03:54AM (#47931601)
Have you tried using Browserless/similar to scrape around tricky hosts?
Re:Scraping 241 UK council planning portals – 2.6M decisions... by mebkorea on Tuesday April 28, 2026 @04:05AM (#47931681)
No, I haven't tried Browserless. So far, it has all been from a single residential IP which is probably the bigger issue with Liverpool than the WAF challenge itself. Once I have a valid session cookie I can solve the JS challenge fine, the rate limit is per-IP. Rotating residential proxies (or Browserless behind one) might be the answer... I'm just reluctant at this stage to bite the bullet on the cost for a single (albeit huge) council. Have you used it for similar stuff?