Re:How we learnt to stop worrying and love web scraping
by uhtred on Wednesday September 09, 2020 @10:08AM
(#24421112)
Seems like a lot of interesting sites recognise that I am scraping and block access with a capture. I've no idea how to get around that.
[ Parent
| on HN ]
Re:How we learnt to stop worrying and love web scraping
by gmac on Wednesday September 09, 2020 @10:11AM
(#24421147)
How are you doing it? Automating an actual browser using the developer tools is probably the most under-the-radar way, and also quite nice to work with.
Re:How we learnt to stop worrying and love web scraping
by ivoecpereira on Wednesday September 09, 2020 @12:24PM
(#24422654)
What out for "Devtools detect" :) https://github.com/sindresorhus/devtools-detect
Re:How we learnt to stop worrying and love web scraping
by gmac on Wednesday September 09, 2020 @12:52PM
(#24422965)
This is a good point, and I've noticed this phenomenon on some video sites lately.
Although the caveats for this particular library[1] imply enough false positives and false negatives that it seems mostly useless. Sites that take this seriously must be doing something smarter.
[1] "Doesn't work if DevTools is undocked and will show false positive if you toggle any kind of sidebar."
Re:How we learnt to stop worrying and love web scraping
by umaar on Wednesday September 09, 2020 @11:03AM
(#24421796)
I've put an example here, for doing it in Node.js: https://github.com/umaar/learn-browser-testing/tree/master/2...