News for hackers. Stuff that matters.
Re:Wikipedia-IPFS: An exploration to host Wikipedia in IPFS by jokoon on Sunday May 10, 2020 @04:47AM (#23131211)
I recently tried to use IPFS to share 20 gigabytes of data on a local network, using ipfs-desktop.

Apparently it hashes the entire file, meaning it will read it entirely, so I killed it because it took way too long. Apparently there are other methods for fingerprinting large files, like reading a limited set of chunks. I asked on IRC but I did not get a thorough answer.

Re:Wikipedia-IPFS: An exploration to host Wikipedia in IPFS by imhoguy on Sunday May 10, 2020 @05:18AM (#23131334)
To share files in a local network FTP or SMB/CIFS may be better suited.

IPFS is content addressable storage by design on entire files. Reading limited set of chunks poses some problem regarding data integrity and deduplication colisions. One bit flip in JPG may destroy the entire image - thankfuly with currently used hash functions the right and broken files will have different "fingerprint". One way hash functions have extremely low probability of collision with full file scan. However when you skip part of files the probalibity of collision is roughly equal to count of bytes not scaned divided by total file size.

Re:Wikipedia-IPFS: An exploration to host Wikipedia in IPFS by toomuchtodo on Sunday May 10, 2020 @02:51PM (#23135005)
It’s a shame modern file systems don’t generate and store common hash algorithms as file metadata.
Re:Wikipedia-IPFS: An exploration to host Wikipedia in IPFS by imhoguy on Sunday May 10, 2020 @03:54PM (#23135566)
Some file systems like ZFS, btrfs support checksum on block level. However file level checksum is not trivial with random write file system. Imagine 1 byte is changed in 20GB file - that would require full file scan to update the checksum. Recalculating a file checksum from all block checksums could be a solution however far from any standard.
Re:Wikipedia-IPFS: An exploration to host Wikipedia in IPFS by brokenmachine on Sunday May 10, 2020 @09:58PM (#23138171)
Wouldn't a merkle tree, like torrents or dat use, avoid this? You'd only have to generate a hash of the changed chunk, then it's really low cost to update the root hash.

https://datprotocol.github.io/book/ch01-02-merkle-tree.html

Re:Wikipedia-IPFS: An exploration to host Wikipedia in IPFS by imhoguy on Monday May 11, 2020 @08:03AM (#23141197)
Yep, my last sentence covered also Markle tree.
Re:Wikipedia-IPFS: An exploration to host Wikipedia in IPFS by fit2rule on Sunday May 10, 2020 @06:10AM (#23131516)
I don't understand why you got impatient? Just let it finish and then you can share, easily enough across your entire network.