One of my dream projects as well, sadly it feels a lot harder to crawl the internet these days, as others have said around here as well.

What are some good practices these days to ensure a good crawl/scrape? Invest in proxies, preferably residential?