Re:Web Scraping 101 with Python
by mikece on Wednesday February 10, 2021 @12:14PM
(#26091934)
Aside from the Beautiful Soup library, is there something about Python that makes it a better choice for web scraping than languages such as Java, JavaScript, Go, Perl or even C#?
[ Parent
| on HN ]
Don't know about large scales, but just today I threw together a script using selenium, imported pandas to mangle the scraped data and quickly exported to json. For quick and dirty, possibly one-off jobs like that, Python is a great choice.
Re:Web Scraping 101 with Python
by jbergstroem on Wednesday February 10, 2021 @12:29PM
(#26092128)
The Scrapy library written in Python (https://scrapy.org) is excellent for writing and deploying scrapers.
Re:Web Scraping 101 with Python
by lemagedurage on Wednesday February 10, 2021 @12:22PM
(#26092035)
I think Python makes sense, at least for the prototyping phase. There's a lot of trial and error involved, and Python is quick to write.
Re:Web Scraping 101 with Python
by freedomben on Wednesday February 10, 2021 @12:24PM
(#26092061)
I find javascript (node) to be best suited to web scraping personally. Using the same language to scrape/process as you use to develop those interfaces seems most natural.
Re:Web Scraping 101 with Python
by isbvhodnvemrwvn on Wednesday February 10, 2021 @03:22PM
(#26094201)
Especially with stuff like Puppeteer which allows you to execute JS in context of the browser (which admittedly can lead to weird bugs as the functions are serialized and lose context)
I'm using this exact strategy to scrape content directly from DOM using APIs like document.querySelectorAll. You can use the same code in both headless browser clients like Puppeteer or Playwright and DOM clients like cheerio or jsdom (assuming you have a wrapper over document API). Depending on the way a web page was fetched (opened in a browser tab or fetched via nodejs http/https requests), ExtractHtmlContentPlugin, ExtractUrlsPlugin use different DOM wrappers (native, cheerio, jsdom) to scrape the content.
Re:Web Scraping 101 with Python
by monkeybutton on Wednesday February 10, 2021 @12:25PM
(#26092079)
I like python for the ease of use and scraping is I/O bound anyways so there's no pressure to switch to a more performant language.
Re:Web Scraping 101 with Python
by RhodesianHunter on Wednesday February 10, 2021 @12:29PM
(#26092136)
I'd say that really depends on your scale and what you're doing with the content you scrape.
In my experience with large scale scraping you're much better off using something like Java where you can more easily have a thread pool with thousands of threads (or better yet, Kotlin coroutines) handling the crawling itself and a *NUM CORES thread pool handling CPU bound tasks like parsing.
Re:Web Scraping 101 with Python
by monkeybutton on Wednesday February 10, 2021 @12:57PM
(#26092534)
Could you give a ballpark figure for what you mean by large scale scraping? I've only worked on a couple projects, one was a broad (100K to 500K domains) and shallow (root + 1 level of page depth, also with a low cap on the number of children pages). The other just a single domain but scraping around 50K pages from it.
I would say millions of domains regularly. That's where the pricing of most 'scraping services' falls down too compared to just doing it yourself.
Re:Web Scraping 101 with Python
by RhodesianHunter on Wednesday February 10, 2021 @02:15PM
(#26093431)
My experience was with e-commerce scraping. Not many domains, but a massive catalogue.