News for hackers. Stuff that matters.
Re:Willow Protocol by ComputerGuru on Wednesday January 17, 2024 @12:28PM (#39030704)
Willow solves the biggest problem I have always had with IPFS: it’s content addressable, which is nice for specific things but not generic enough to make the protocol actually practical and usable in the real world outside of specific use cases. (Namely you can’t update resources or even track related updates to resources.)

(Mathematically, a name-addressable system is actually a superset of content-addressable systems as you can always use the hash of the content as the name itself.)

Re:Willow Protocol by nayuki on Wednesday January 17, 2024 @02:12PM (#39032035)
> a name-addressable system is actually a superset of content-addressable systems as you can always use the hash of the content as the name itself

It's a superset in that sense but not a superset in another sense.

In a content-addressable system, if I post a link to another piece of content by hash, then no one can ever substitute a different piece of content. Like, if I reference the hash of a news article, no one can edit that article after the fact without being detected. This is a super-useful feature of CAS that is not a feature of NAS. Other implications:

* I can review a piece of software, deem that it's not malware in my opinion, and link to the software by hash. No one can substitute a piece of malware without detection.

* Suppose you get a link from a trusted source. Now you can download a copy of the underlying content from any untrusted source, without a care about authentication or trusted identities. This describes BitTorrent.

Re:Willow Protocol by ComputerGuru on Thursday January 18, 2024 @09:16PM (#39050839)
It’s a protocol, not an implementation. If you say your implementation of the protocol says objects are defined by their hash, then your implementation can also assert that the hash of the retrieved file matches the hash you looked it up with^ (which you should do in your library/app regardless of what the protocol says will be returned, though ideally your protocol would define some sort of merkle tree or whatever to verify the hash piecemeal).

^ which you, by definition, already have

Re:Willow Protocol by rklaehn on Wednesday January 17, 2024 @03:37PM (#39033100)
To be fair, IPFS does offer not just content addressing but also a mechanism for mutability with IPNS. You can think of a willow namespace (or iroh document) as a key value store of IPNS entries.

The problem with IPNS is that the performance is... not great... to put it politely, so it is not really an useful primitive to build mutability.

You end up building your own thing using gossip, at which point you are not really getting a giant benefit anymore.

Re:Willow Protocol by unshavedyak on Wednesday January 17, 2024 @05:12PM (#39034283)
Is the performance a critical design flaw or just an implementation issue?
Re:Willow Protocol by tadfisher on Wednesday January 17, 2024 @11:58PM (#39038056)
Imagine if DNS supplied every URL and not just domain names. You need some mechanism to propagate resource changes. IPNS has two practical mechanisms: a global DHT that takes time to propagate, and a pub/sub that requires peers to actively subscribe to your changes.
Re:Willow Protocol by anacrolix on Thursday January 18, 2024 @05:49AM (#39040268)
btlink does DNS per domain name, which you could argue is a sweet spot between too many queries and being too broad. at least in the case of the web, it works nicely.
Re:Willow Protocol by rklaehn on Thursday January 18, 2024 @03:45AM (#39039468)
Difficult to answer.

IPNS uses the IPFS kademlia DHT, which has some performance problems that you can argue are fundamental.

For solving a similar problem with iroh, we would use the bittorrent mainline DHT, which is the largest DHT in existence and has stood the test of time - it still exists despite lots of powerful entities wanting it to go away.

It also generally has very good performance and a very minimalist design.

There is a rust crate to interact with the mainline DHT, https://crates.io/crates/mainline , and a more high level idea to use DHTs as a kind of p2p DNS, https://github.com/nuhvi/pkarr

Re:Willow Protocol by anacrolix on Thursday January 18, 2024 @05:49AM (#39040270)
It's a design flaw.
Re:Willow Protocol by wscott on Thursday January 18, 2024 @01:27PM (#39045557)
Design flaw. In IPFS every piece of data (even every chuck of large files) is globally indexable on the same namespace. You need namespaces and/or a path to restrict yourself to just the subset of peers that might actually have the data.

It would be possible to add a layer on top of IPFS to include some context with every hash lookup so the search can be more focused, but the original design just assumed it was ok to do a log2 search for every chunk.

Re:Willow Protocol by rklaehn on Friday January 19, 2024 @04:48AM (#39053554)
That is not a problem specific to IPNS though. Using a DHT for something like IPNS is fine. Publishing roots of large data sets is also fine(ish).

Using it to publish every tiny chunk of a large file is a horrible idea. It leads to overwhelming traffic.

If you publish a few TB of data, due to the randomness of the DHT xor metric you have to basically talk to every node on the network. Add to that the fact that establishing a tcp libp2p connection is much more heavyweight than sending a single UDP packet like in the bittorrent mainline DHT, and you are basically screwed.

In iroh we don't publish at all by default. But if you have to use a DHT, the fact that we have a single hash for arbitrary large files due to blake3 verified streaming helps a lot.

You still get verified range access.