edit: yep here it is https://doi.org/10.1145/3184558.3191636
We've been around for about a decade. IBM watson used us as their social data provider during Jeopardy. We provide data to tons of companies and you're probably using our services - just that it's not obvious where we're used since it's SaaS B2B and not B2C.
We're not free but the primary reason we exist is that other vendors charge borderline extortionate pricing and I fundamentally believe that the web MUST remain open.
We've also been providing data for very affordable pricing to researchers for more than a decade.
Search for us as Spinn3r under Google Scholar (our previous name) and we have hundreds and hundreds of PhDs who have access to our data.
We do charge for research usage now but it's very very very affordable.
The entire point is that we're trying to enable innovation.
Yet the company first mentioned does it for free, lol:
I've checked Datastreamer.io for 5 seconds, I don't see any link to their repo. If not "open source" then what does "open" mean?
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize.
Your comment makes zero sense in this context, because it's just marketing.
> we're trying to enable innovation.
You're trying to make profit, like every other company in the world and that's OK.
A site that gets lots of links quickly (and is therefore important) will likely garner them from sites you are already frequently visiting.
There are a significant number of crawlers out there that don't respect robots.txt. The usual response to them isn't to roll over dead, it's to get CloudFlare (on the technological end) and/or sic the lawyers on them (for CFAA, IP, or ToS violations).
While you say "news" really that covers any information about current-ish events. It's not just "what happened today" but background on things like the Muller report right now.
Any technology release, or update.
Reviews of any hardware or software.
Information about security vulnerabilities.
Film reviews.
Game reviews.
Book reviews.
New scientific publications.
Although, that said, with Google personalising search results the top result is very likely to be the user's preferred site anyway. We can't have people seeing outside their filter bubble after all.
What you’re describing is just news on the latest updates representing a small slice of the market. Making the remander far from useless.
> What is Common Crawl?
> Common Crawl is a 501(c)(3) non-profit organization dedicated to providing a copy of the internet to internet researchers, companies and individuals at no cost for the purpose of research and analysis.
> What can you do with a copy of the web?
> The possibilities are endless, but people have used the data to improve language translation software, predict trends, track the disease propagation and much more.
> Can’t Google or Microsoft just do that?
>Our goal is to democratize the data so everyone, not just big companies, can do high quality research and analysis.
Also DuckDuckGo founder Gabriel Weinberg expressed the sentiment that the index should be separate from the search engine many years ago:
> Our approach was to treat the “copy the Internet” part as the commodity. You could get it from multiple places. When I started, Google, Yahoo, Yandex and Microsoft were all building indexes. We focused on doing things the other guys couldn’t do. [2]
From what I remember reading once DuckDuckGo doesn't use Common Crawl though.
[1] https://commoncrawl.org/
[2] https://www.japantimes.co.jp/news/2013/07/28/business/duckdu...