Regardless. You should still respect robots.txt..
> comparing publicly listed product prices across e-commerce sites
Those prices and information is for the public viewers, the reason why some people have ROBOTS.txt for example is to reduce the traffic load that slop crawlers generate. The bandwidth is not free so why would you assume to ignore their ROBOTS.txt when you're not footing the bill ?
Re:
by andrew_zhong on Thursday March 26, 2026 @02:55PM
(#47534258)
[Update]] I will replace the stealth browser with plain playwright and remove anti-bot as a feature.
Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we should make that clearer in the README.