Re:How to Crawl the Web with Scrapy
by getcrunk on Monday September 13, 2021 @08:39PM
(#28518766)
Was rate limiting not a concern? How often did you run it? I'm sure you had to keep some state too?
[ Parent
| on HN ]
Their website was pretty trivial, but I beefed up the script to check for failure status code and also notify me if it wasn't 200.
I ran it every minute.
I had to include a unique uuid to cache bust, but other than that there wasn't any state.
I wrote about it here https://plainice.com/webscrape-notifications