News for hackers. Stuff that matters.

A comment on /story/47526486. Parent comment

Re: by andrew_zhong on Thursday March 26, 2026 @03:27AM (#47527589)
We do respect robots.txt production - also scraping browser providers like BrightData enforces that.

I will add a PR to enforce robots.txt before the actual scraping.

[ Parent | on HN ]
Re: by plastic041 on Thursday March 26, 2026 @03:54AM (#47527736)
How can people believe that you are respecting bot detection in production when your software's README says it can "Avoid detection with built-in anti-bot patches"?
[ Parent | on HN ]
Re: by andrew_zhong on Thursday March 26, 2026 @02:54PM (#47534243)
I hear you loud and clear - will replace the stealth browser with plain playwright and remove anti-bot as a feature.
[ Parent | on HN ]