Re:Lightpanda: The headless browser designed for AI and auto...
by slt2021 on Friday January 24, 2025 @03:00PM
(#42816433)
it is trivial to spoof user-agent, if you want to stop a motivated scraper, you need a different solution that exploits the fact that robots use headless browser
[ Parent
| on HN ]
Re:Lightpanda: The headless browser designed for AI and auto...
by sangnoir on Friday January 24, 2025 @03:12PM
(#42816507)
> it is trivial to spoof user-agent
It's also trivial to detect spoofed user agents via fingerprinting. The best defense against scrapers is done in layers, with user-agent name block as the bare minimum.