News for hackers. Stuff that matters.
Re:Willow Protocol by unshavedyak on Wednesday January 17, 2024 @05:12PM (#39034283)
Is the performance a critical design flaw or just an implementation issue?
Re:Willow Protocol by tadfisher on Wednesday January 17, 2024 @11:58PM (#39038056)
Imagine if DNS supplied every URL and not just domain names. You need some mechanism to propagate resource changes. IPNS has two practical mechanisms: a global DHT that takes time to propagate, and a pub/sub that requires peers to actively subscribe to your changes.
Re:Willow Protocol by anacrolix on Thursday January 18, 2024 @05:49AM (#39040268)
btlink does DNS per domain name, which you could argue is a sweet spot between too many queries and being too broad. at least in the case of the web, it works nicely.
Re:Willow Protocol by rklaehn on Thursday January 18, 2024 @03:45AM (#39039468)
Difficult to answer.

IPNS uses the IPFS kademlia DHT, which has some performance problems that you can argue are fundamental.

For solving a similar problem with iroh, we would use the bittorrent mainline DHT, which is the largest DHT in existence and has stood the test of time - it still exists despite lots of powerful entities wanting it to go away.

It also generally has very good performance and a very minimalist design.

There is a rust crate to interact with the mainline DHT, https://crates.io/crates/mainline , and a more high level idea to use DHTs as a kind of p2p DNS, https://github.com/nuhvi/pkarr

Re:Willow Protocol by anacrolix on Thursday January 18, 2024 @05:49AM (#39040270)
It's a design flaw.
Re:Willow Protocol by wscott on Thursday January 18, 2024 @01:27PM (#39045557)
Design flaw. In IPFS every piece of data (even every chuck of large files) is globally indexable on the same namespace. You need namespaces and/or a path to restrict yourself to just the subset of peers that might actually have the data.

It would be possible to add a layer on top of IPFS to include some context with every hash lookup so the search can be more focused, but the original design just assumed it was ok to do a log2 search for every chunk.

Re:Willow Protocol by rklaehn on Friday January 19, 2024 @04:48AM (#39053554)
That is not a problem specific to IPNS though. Using a DHT for something like IPNS is fine. Publishing roots of large data sets is also fine(ish).

Using it to publish every tiny chunk of a large file is a horrible idea. It leads to overwhelming traffic.

If you publish a few TB of data, due to the randomness of the DHT xor metric you have to basically talk to every node on the network. Add to that the fact that establishing a tcp libp2p connection is much more heavyweight than sending a single UDP packet like in the bittorrent mainline DHT, and you are basically screwed.

In iroh we don't publish at all by default. But if you have to use a DHT, the fact that we have a single hash for arbitrary large files due to blake3 verified streaming helps a lot.

You still get verified range access.