Re:Lessons learned scraping 100B product pages
by pdimitar on Tuesday July 10, 2018 @03:07PM
(#17501163)
Then I think we have a misunderstanding of terms. To me "multicore" == "single process, many threads". Apologies for the confusion.
It seems that now we are both on the same page. Single process & many threads are problematic for GIL languages and that's why I gave up using Ruby for scrapers. GIL languages can work very well for the URL downloading part though.
If you were to use multithreading instead, you would generally have a problem if you were doing non-I/O work.