News for hackers. Stuff that matters.
Re:Common Crawl by gillesjacobs on Friday March 26, 2021 @02:40PM (#26595416)
Blockchain storage is going to cost you a pretty penny if you were to store all of Common Crawls pentabytes, so not very feasible.
Re:Common Crawl by kevincox on Saturday March 27, 2021 @09:19AM (#26602085)
IPFS doesn't require any blockchain.

In fact someone could easily setup some IPFS nodes that fetch the data from the current host if requested over IPFS.[1] This way people could access it via IPFS and provide an alternate mirror of the data.

[1] https://github.com/ipfs/go-ipfs/blob/master/docs/experimenta...

The main benefits here would be

- Even if the source is unavailable there may be other copies on IPFS which would be transparently used.

- There may be some performance benefits in rare cases.

- If you are accessing this on a bunch of machines your IPFS gateways would handle downloading the source once, then automatically using the local copy from inside your network.

The maindownside is that if Amazon is donating their resources why bother with IPFS?

Re:Common Crawl by psKama on Friday March 26, 2021 @03:26PM (#26595922)
That's not correct. When it comes to storage and transfer, blockchain alternatives are fraction of the cost of Amazon. For example Sia Skynet is offering $5/month/TB[1] storage. If you skip Skynet and run your own Sia node the price can even go lower to $2/month/TB basing on the market conditions.

[1] https://blog.sia.tech/announcing-skynet-premium-plans-faster...

Re:Common Crawl by coder543 on Friday March 26, 2021 @05:51PM (#26597402)
Amazon is hosting the Common Crawl on S3 for free, so... yes, $2/month/TB is a lot more expensive.
Re:Common Crawl by gillesjacobs on Friday March 26, 2021 @06:51PM (#26597946)
It seems that at least on Sia's plans, you can maximally host 20TB for 80$/month, not even a tenth of a monthly common crawl.

Of course Sia's Skynet are package deals right now and I guess they're currently bootstrapping the network with users. Filecoin has no operational storage yet. Storj quotes 10$/Terabyte/month [1] so that will come out expensive.

1. https://www.storj.io/blog/2019/11/announcing-pioneer-2-and-t...

Re:Common Crawl by gloriousternary on Friday March 26, 2021 @04:16PM (#26596428)
Moreso than Amazon? From my (limited) experience blockchain storage solutions are often less expensive, although I've never worked with petabytes of data so maybe it's different on that scale.