News for hackers. Stuff that matters.
Re:How to Crawl the Web with Scrapy by artembugara on Monday September 13, 2021 @04:06PM (#28516132)
We have to crawl about 60-80k news websites per day [0].

I've spent about 1 month to test how scrapy could be a fit for our purposes. And, quite surprisingly, it was hard to design a distributed web crawler. Scrapy is great for those in-the-middle tasks where you need to crawl a bit + process data on the go.

We ended up just using requests to crawl the web. Then post-process the web pages in the next step.

Many thanks to Zyte [1] (ex-ScrapingHub) for open-sourcing so many wonderful tools for us. I've spoke to Zyte's CEO, and was really fascinated how he still being a dev person while running such a big company.

[0] https://newscatcherapi.com/news-api [1] https://www.zyte.com/

Re:How to Crawl the Web with Scrapy by zepearl on Tuesday September 14, 2021 @05:38PM (#28531713)
> I've spent about 1 month to test how scrapy could be a fit for our purposes. And, quite surprisingly, it was hard to design a distributed web crawler. Scrapy is great for those in-the-middle tasks where you need to crawl a bit + process data on the go.

I agree (but in my case I needed 3 months to understand that :) ).

I did start with plain "Scrapy" and it did work fine (maybe some parameters could be explained a bit better especially the ones involving concurrency) and it has been in general very helpful to initially understand some concepts related to web scraping.

But now, after having rewritten multiple parts like 50 times to be able to handle most of what can happen, having split the workload into a multi-staged pipeline involving different specialized programs, having implemented custom code to distribute domains to be scraped to dedicated subprocesses (to greatly improve performance), and much more... I am using Scrapy basically to only download the raw bytes and to take care of what's mentioned in "robots.txt" => my next step will most probably be to replace it (now I don't like a lot having it as a dependency for my program, I therefore started thinking that such a complicated lib is a bit a "risk") with "requests" and something that can help to interpret "robots.txt".

In my case ("wide" Internet scan involving so far ~8M webpages distributed over ~500K domains) I think that it might have been better to start straight with just using "requests".

Re:How to Crawl the Web with Scrapy by m_ke on Monday September 13, 2021 @09:07PM (#28518949)
Even with scrapy it's usually best to to crawl and store full html that can be scraped offline. It's too easy to miss something when scraping on the fly or drop data that might be useful in the future.
Re:How to Crawl the Web with Scrapy by ramraj07 on Tuesday September 14, 2021 @07:51AM (#28523288)
How do you deal with these god awful SPAs though, when you just do requests..
Re:How to Crawl the Web with Scrapy by TechBro8615 on Tuesday September 14, 2021 @11:45AM (#28526518)
SPAs tend to be some of the easiest sites to scrape (at least if you’re building targeted scrapers), because they come with a production-ready API out of the box.

You can just watch the network tab of dev tools to know what endpoints to hit. Same applies to mobile apps – before you reach for BeautifulSoup, maybe check to see if the website has a mobile app too. It’s usually worth setting up a mitmproxy to see what API you might be able to scrape.

Re:How to Crawl the Web with Scrapy by artembugara on Tuesday September 14, 2021 @07:52AM (#28523302)
We do it as a part of post-processing, we'd have to make one more call.

Quite annoying though

Re:How to Crawl the Web with Scrapy by 1vuio0pswjnm7 on Monday September 13, 2021 @07:16PM (#28518145)
Does newscatcherapi provide a list of the 60-80K news sites so the customer knows what sources are actually being searched. Or even just way to determine if site X is among the 60-80K being searched.
Re:How to Crawl the Web with Scrapy by artembugara on Monday September 13, 2021 @07:26PM (#28518224)
Second one — yes. All sources, per request.

But as a general rule, we crawl all we could/know which is a news website.

Can add any niche/industry-specific websites on request.

Re:How to Crawl the Web with Scrapy by jcun4128 on Monday September 13, 2021 @06:37PM (#28517820)
> We have to crawl about 60-80k news websites per day [0]

Can't even imagine that number... different languages or something?

Re:How to Crawl the Web with Scrapy by artembugara on Monday September 13, 2021 @06:43PM (#28517876)
Yeah, plus many of websites are quite niche news (construction news, for example)
Re:How to Crawl the Web with Scrapy by adamqureshi on Monday September 13, 2021 @04:10PM (#28516177)
ok so i can just hire zyte to build me a custom scraper?
Re:How to Crawl the Web with Scrapy by artembugara on Monday September 13, 2021 @04:18PM (#28516307)
well, I think it is the cheapest & the fastest way, tbh.
Re:How to Crawl the Web with Scrapy by adamqureshi on Monday September 13, 2021 @04:22PM (#28516357)
Thank you. The other site is also very interesting. I am working on this MVP and its news aggregator type site for a NICHE product. So i need to aggregate news for a brand from maybe 10-20 blogs and list the URL. thank you for sharing both. I'll reach out to them.
Re:How to Crawl the Web with Scrapy by artembugara on Monday September 13, 2021 @04:24PM (#28516386)
I'm the co-founder of the other one, we could help you with your task.

Feel free to contact me. artem [at] newscatcherapi.com

Re:How to Crawl the Web with Scrapy by adamqureshi on Monday September 13, 2021 @04:38PM (#28516575)
Oh awesome! emailing you now. Thank you.