News for hackers. Stuff that matters.
Re:How to Crawl the Web with Scrapy by r_singh on Monday September 13, 2021 @03:05PM (#28515392)
I've used Scrapy extensively for writing crawlers.

There's a lot of good things like not having to worry about storage backends, request throttling (random seconds between requests), the ability to run parallel parsers easily. There is also a lot of open source middleware to help with things like retrying requests with proxies and rotating user agents.

However, like any battery included framework it has downsides in terms of flexibility.

In most cases requests and lxml should be enough to crawl the web.

Re:How to Crawl the Web with Scrapy by ducktective on Monday September 13, 2021 @03:58PM (#28516029)
> In most cases requests and lxml should be enough to crawl the web.

Don't mind my `curl | pup xmlstarlet grep(!!)`s... Nothing to see here...

Re:How to Crawl the Web with Scrapy by amozoss on Monday September 13, 2021 @04:09PM (#28516173)
My brother-in-law had just finished his pilot training and was trying to apply for a job as teacher to continue his training.

However, the jobs were first come, first serve so he was waking up at 4 am and constantly refreshing for hours trying to be the first one.

When I heard about it, I quickly whipped up a `curl | grep && send_notif` (used pushback.io for notifs) and it helped him not have to worry so much.

When a new job posting finally came along he was the first in line and got the job :)

Re:How to Crawl the Web with Scrapy by getcrunk on Monday September 13, 2021 @08:39PM (#28518766)
Was rate limiting not a concern? How often did you run it? I'm sure you had to keep some state too?
Re:How to Crawl the Web with Scrapy by amozoss on Monday September 13, 2021 @10:57PM (#28519674)
Their website was pretty trivial, but I beefed up the script to check for failure status code and also notify me if it wasn't 200.

I ran it every minute.

I had to include a unique uuid to cache bust, but other than that there wasn't any state.

I wrote about it here https://plainice.com/webscrape-notifications

Re:How to Crawl the Web with Scrapy by aynyc on Monday September 13, 2021 @04:06PM (#28516135)
If you are just doing one or two pages, say you want to get weather for your location, then requests is sufficient. But if you want to do many pages where you might want to scan and follow, requests gets tedious very quickly.
Re:How to Crawl the Web with Scrapy by r_singh on Monday September 13, 2021 @04:33PM (#28516503)
If you’re a web developer not really, rather than worrying about storage backendes, spiders, yielding and managing loops and items, you could just host a DRF or Flask API with your scrapers (written in Requests+lxml) initiated with an API request.

I guess it’s a matter of preference