News for hackers. Stuff that matters.
Re:Web Scraping 101 with Python by monkeybutton on Wednesday February 10, 2021 @12:25PM (#26092079)
I like python for the ease of use and scraping is I/O bound anyways so there's no pressure to switch to a more performant language.
Re:Web Scraping 101 with Python by RhodesianHunter on Wednesday February 10, 2021 @12:29PM (#26092136)
I'd say that really depends on your scale and what you're doing with the content you scrape.

In my experience with large scale scraping you're much better off using something like Java where you can more easily have a thread pool with thousands of threads (or better yet, Kotlin coroutines) handling the crawling itself and a *NUM CORES thread pool handling CPU bound tasks like parsing.

Re:Web Scraping 101 with Python by monkeybutton on Wednesday February 10, 2021 @12:57PM (#26092534)
Could you give a ballpark figure for what you mean by large scale scraping? I've only worked on a couple projects, one was a broad (100K to 500K domains) and shallow (root + 1 level of page depth, also with a low cap on the number of children pages). The other just a single domain but scraping around 50K pages from it.
Re:Web Scraping 101 with Python by tluyben2 on Wednesday February 10, 2021 @01:51PM (#26093204)
I would say millions of domains regularly. That's where the pricing of most 'scraping services' falls down too compared to just doing it yourself.
Re:Web Scraping 101 with Python by RhodesianHunter on Wednesday February 10, 2021 @02:15PM (#26093431)
My experience was with e-commerce scraping. Not many domains, but a massive catalogue.