News for hackers. Stuff that matters.
Re:How we learnt to stop worrying and love web scraping by gmac on Wednesday September 09, 2020 @10:11AM (#24421147)
How are you doing it? Automating an actual browser using the developer tools is probably the most under-the-radar way, and also quite nice to work with.

See: https://github.com/jawj/web-scraping-for-researchers

Re:How we learnt to stop worrying and love web scraping by ivoecpereira on Wednesday September 09, 2020 @12:24PM (#24422654)
What out for "Devtools detect" :) https://github.com/sindresorhus/devtools-detect
Re:How we learnt to stop worrying and love web scraping by gmac on Wednesday September 09, 2020 @12:52PM (#24422965)
This is a good point, and I've noticed this phenomenon on some video sites lately.

Although the caveats for this particular library[1] imply enough false positives and false negatives that it seems mostly useless. Sites that take this seriously must be doing something smarter.

[1] "Doesn't work if DevTools is undocked and will show false positive if you toggle any kind of sidebar."