News for hackers. Stuff that matters.
Re:The Web is missing an essential part of infrastructure: a... by Sir_Cmpwn on Sunday April 21, 2019 @10:21PM (#19715903)
This doesn't make any sense. You talk about open data but yours is the opposite. You're just another commercial data hoarder, please don't act like you're not.
Re:The Web is missing an essential part of infrastructure: a... by sytelus on Monday April 22, 2019 @12:05AM (#19716444)
You are mistaking between free and open. You can be open without being free. Maintaining web index is extremely expensive. Imagine storing most of the web on your own servers and serving it. Someone has to pay bills for all those disk space and bandwidth. I don’t think web index would ever be free (unless storage, compute and bandwidth were free) but having at reasonably priced is a very good thing. I would hope these indices are available on AWS, Azure etc where people can just use it with cloud compute and pay per use.
Re:The Web is missing an essential part of infrastructure: a... by jonathanstrange on Monday April 22, 2019 @03:51AM (#19717243)
Easy to test, though. If they were open, you could download their entire data set under some permissive license. If you can't then they are not open.
Re:The Web is missing an essential part of infrastructure: a... by profalseidol on Monday April 22, 2019 @02:15AM (#19716920)
> I don’t think web index would ever be free

Yet the company first mentioned does it for free, lol:

https://commoncrawl.org/

I've checked Datastreamer.io for 5 seconds, I don't see any link to their repo. If not "open source" then what does "open" mean?

Re:The Web is missing an essential part of infrastructure: a... by ksahin on Monday April 22, 2019 @03:35AM (#19717189)
Commoncrawl is not a company, it's a non-profit. Open means you can access the data, there is no assumption about the data being free or not.
Re:The Web is missing an essential part of infrastructure: a... by warent on Monday April 22, 2019 @04:01AM (#19717274)
What? It's a nonprofit organization engaging in nonprofit business. Any organization that engages in business is a "company." Common Crawl is a company. Your comment isn't accurate and it doesn't address the parent's comment.