That’s exactly what we use for a competing product. The difficulty is less in the crawling, and more in the scheduling and difference visualization (including filters). Presenting salient results without a ton of false positives is hard work.