Re:How to crawl a quarter billion webpages in 40 hours (2012)
by arek_nawo on Thursday June 15, 2023 @08:04AM
(#36339276)
Which way to go will depend on your use-case and what websites you want to scrape.
For occasional web crawling, headless browser is great as it's easy to set up and you're almost guaranteed to get the data you need.
For frequent or large-scale crawling, it's a different story. Surely you can implement something with just HTTP library, but you'll need to test it throughly and make some research before hand. That said, most scraped, content-heavy websites use either static HTML or SSR, in which case you can use HTTP no problem.
For occasional web crawling, headless browser is great as it's easy to set up and you're almost guaranteed to get the data you need.
For frequent or large-scale crawling, it's a different story. Surely you can implement something with just HTTP library, but you'll need to test it throughly and make some research before hand. That said, most scraped, content-heavy websites use either static HTML or SSR, in which case you can use HTTP no problem.