News for hackers. Stuff that matters.

netborn

Re:Ask HN: Why did your startup fail? on Monday May 27, 2019 @01:07PM (#20023605)
Whereas this may be a common reason for failure, we failed because we were all engineers. Nobody with management/sales experience on the team!
There are two entities trying to pull this off:

Common Crawl (non-profit): Stores regular, broad, monthly crawls as WARC files. Provides a separate index that can be used to look data up (no a fulltext index though). Used mostly in academia.

Mixnode (for-profit): Regularly crawls the web and lets users write SQL queries against the data. Not sure who the primary users are since it's in private beta.

There are some search engine APIs, but I don't think the conflict of interest would allow for cost-effective large-scale access and pricing...