Re:
by plastic041 on Thursday March 26, 2026 @03:23AM
(#47527562)
robots.txt is the most basic access restrictions and it doesn't even read it, while faking itself as human[0]. It is about bypassing access restrictions.
[0]: https://github.com/lightfeed/extractor/blob/d11060269e65459e...
Regardless. You should still respect robots.txt..
Re:
by andrew_zhong on Thursday March 26, 2026 @03:27AM
(#47527589)
We do respect robots.txt production - also scraping browser providers like BrightData enforces that.
I will add a PR to enforce robots.txt before the actual scraping.
Re:
by plastic041 on Thursday March 26, 2026 @03:54AM
(#47527736)
How can people believe that you are respecting bot detection in production when your software's README says it can "Avoid detection with built-in anti-bot patches"?
Re:
by andrew_zhong on Thursday March 26, 2026 @02:54PM
(#47534243)
I hear you loud and clear - will replace the stealth browser with plain playwright and remove anti-bot as a feature.
> It's not about bypassing access restrictions.
Yes. It is. You've just made an arbitrary choice not to define it as such.
Re:
by andrew_zhong on Thursday March 26, 2026 @03:31AM
(#47527617)
I will add a PR to enforce robots.txt before the actual scraping.
Or just follow web standards and define and publish your User-Agent header, so that people can block that as needed.
You're creating the wrong kind of value. I really hope your company fails, as its success implies a failure of the web in general.
I wish you the best success outside of your current endeavour.
Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we should make that clearer in the README.