News for hackers. Stuff that matters.
Re:Ask HN: How to Web Scrape in 2020? by marcell on Sunday March 29, 2020 @03:22AM (#22717066)
scrapy for python is pretty good, check it out.

In most cases getting banned is the big issue. The bigger the site, the more advanced their bot detection is. You can use luminato.io to get residential and mobile IP's, but it's pricey.

Some sites will also obfuscate the DOM, ie. removing classnames and ID's, which complicates the data extraction.

http://scrapinghub.com/ has a paid "do it for me" service, which may be an option depending on your budget.

Re:Ask HN: How to Web Scrape in 2020? by austincheney on Sunday March 29, 2020 @07:17PM (#22723644)
Here is a tiny DOM walking script that evaluates all text semantics in a page demonstrating that you don’t need identifiers in the code.

https://github.com/prettydiff/semanticText