Re:The Web is missing an essential part of infrastructure: a...
by sytelus on Monday April 22, 2019 @12:05AM
(#19716444)
You are mistaking between free and open. You can be open without being free. Maintaining web index is extremely expensive. Imagine storing most of the web on your own servers and serving it. Someone has to pay bills for all those disk space and bandwidth. I don’t think web index would ever be free (unless storage, compute and bandwidth were free) but having at reasonably priced is a very good thing. I would hope these indices are available on AWS, Azure etc where people can just use it with cloud compute and pay per use.
[ Parent
| on HN ]
Re:The Web is missing an essential part of infrastructure: a...
by jonathanstrange on Monday April 22, 2019 @03:51AM
(#19717243)
Easy to test, though. If they were open, you could download their entire data set under some permissive license. If you can't then they are not open.
Re:The Web is missing an essential part of infrastructure: a...
by profalseidol on Monday April 22, 2019 @02:15AM
(#19716920)
> I don’t think web index would ever be free
Yet the company first mentioned does it for free, lol:
I've checked Datastreamer.io for 5 seconds, I don't see any link to their repo. If not "open source" then what does "open" mean?
Re:The Web is missing an essential part of infrastructure: a...
by ksahin on Monday April 22, 2019 @03:35AM
(#19717189)
Commoncrawl is not a company, it's a non-profit. Open means you can access the data, there is no assumption about the data being free or not.
Re:The Web is missing an essential part of infrastructure: a...
by warent on Monday April 22, 2019 @04:01AM
(#19717274)
What? It's a nonprofit organization engaging in nonprofit business. Any organization that engages in business is a "company." Common Crawl is a company. Your comment isn't accurate and it doesn't address the parent's comment.