Re:Lessons learned scraping 100B product pages
by pdimitar on Tuesday July 10, 2018 @03:07PM
(#17501163)
Then I think we have a misunderstanding of terms. To me "multicore" == "single process, many threads". Apologies for the confusion.
It seems that now we are both on the same page. Single process & many threads are problematic for GIL languages and that's why I gave up using Ruby for scrapers. GIL languages can work very well for the URL downloading part though.
It seems that now we are both on the same page. Single process & many threads are problematic for GIL languages and that's why I gave up using Ruby for scrapers. GIL languages can work very well for the URL downloading part though.