Willow Protocol

Re:Willow Protocol by rklaehn on Wednesday January 17, 2024 @09:55AM (#39028532)

if you are looking for something similar to ipfs but a bit more minimalistic and performance oriented, check out iroh https://github.com/n0-computer/iroh .

It is a set of open source libraries for peer to peer networking and content-addressed storage. It is written in rust, but we have bindings to many languages.

One part of iroh is a work in progress implementation of the willow spec. The lower layers include a networking library similar to libp2p and a library for content-addressed storage and replication based on blake3 verified streaming.

Most iroh developers have been active in the ipfs community for many years and have shared similar frustrations... See this talk from me in 2019 :-)

https://youtu.be/Qzu0xtCT-R0?t=169

[ Parent | on HN ]

Re:Willow Protocol by binary132 on Wednesday January 17, 2024 @10:21AM (#39028907)

Interesting — does the Rust crate export a C API then?

[ Parent | on HN ]

Re:Willow Protocol by rklaehn on Wednesday January 17, 2024 @10:49AM (#39029333)

Not officially. We currently have bindings for rust, python, golang and swift.

These were the most asked for bindings (python for ml, golang for networking and swift for ios apps).

We are using uniffi https://mozilla.github.io/uniffi-rs/

Would you need C or C++ bindings?

[ Parent | on HN ]

Re:Willow Protocol by binary132 on Wednesday January 17, 2024 @04:59PM (#39034117)

Ah, I see. Hm. I might be interested in a C API since that could be used in C, C++, and Lua equally well. I really was just wondering what the common implementation between the bindings was since it struck me as unusual that there would be a number of bindings but not C (which is AIUI the only interface besides Rust that Rust can really export.)

[ Parent | on HN ]

Re:Willow Protocol by rklaehn on Thursday January 18, 2024 @01:57AM (#39038814)

So, I am not the one that is doing the bindings, so take this with a grain of salt.

It seems uniffi does create C compatible bindings in order to make bindings for all these other languages. But these are internal bindings that are ugly and not intended to be used externally.

[ Parent | on HN ]

Re:Willow Protocol by binary132 on Thursday January 18, 2024 @09:53AM (#39042359)

Perhaps if an issue were opened describing the necessary steps to provide a fluent and stable C API, staying consistent with the uniffi approach you're using for the other wrappers, then someone enterprising could pick up the ball and run with it. :)

[ Parent | on HN ]

Re:Willow Protocol by ramrunner0xff on Wednesday January 17, 2024 @12:52PM (#39031016)

https://veilid.com/ should also be a great alternative. i haven't had time to use it yet, but it was built to address performance issues with ipfs and allow for both dht style content discovery, but also for direct websocket connections for streaming (and doing that in an anonymous fashion)

[ Parent | on HN ]

Re:Willow Protocol by rklaehn on Wednesday January 17, 2024 @01:56PM (#39031839)

This looks very interesting. They made very similar choices than we (iroh) did. Rust, ed keys, blake3.

They seem to do their own streams, while we are adapting QUIC to a more p2p approach. Also the holepunching approach seems to be different. But I would love to get more details.

[ Parent | on HN ]

Re:Willow Protocol by ramrunner0xff on Wednesday January 17, 2024 @02:37PM (#39032352)

https://yewtu.be/watch?v=Kb1lKscAMDQ

this was the presentation at DC'31. i will also check out iroh! thanks for working in building something in this space, it is much much needed!

[ Parent | on HN ]

Re:Willow Protocol by rklaehn on Wednesday January 17, 2024 @03:14PM (#39032820)

Thanks. This is awesome. I think they are doing more work themselves in terms of crypto, whereas we rely on QUIC+TLS more.

Regarding holepunching, our approach is a bit less pure p2p, but has quite good success rates. We copy the DERP protocol from tailscale.

I am confident that we have a better story regarding handling of large blobs. We don't just use blake3, but blake3 verified streaming to allow for range requests.

Also I wrote my own rust library for blake3 verified streaming that reduces the overhead of the verification data. https://crates.io/crates/bao-tree

I tried to get on their discord at https://veilid.com/discord, but I get an invalid invite. You know a better way to get in touch?

[ Parent | on HN ]

Re:Willow Protocol by ramrunner0xff on Wednesday January 17, 2024 @05:19PM (#39034380)

hmm this is strange, i tried the invite and it worked for me. If you are on fedi, @[email protected] is part of the team.

thanks for the links, i will get in touch personally when i try ir0h :)

[ Parent | on HN ]

Re:Willow Protocol by orthecreedence on Wednesday January 17, 2024 @04:25PM (#39033709)

Hi, I'm super intrigued by Willow and your work on iroh. Do you have any kind of documentation on how iroh deviates from Willow, or what parts of Willow are planned to be implemented vs omitted?

[ Parent | on HN ]

Re:Willow Protocol by rklaehn on Thursday January 18, 2024 @12:02PM (#39044269)

Not yet. We have been busy with other stuff, also the willow spec has been a bit of a moving target until now.

We would like to take our rust willow impl and separate it a bit more from our code base, so that iroh documents are just users of the willow crate.

[ Parent | on HN ]

Re:Willow Protocol by orthecreedence on Thursday January 18, 2024 @01:10PM (#39045241)

That makes sense. I think I might try to really jam through the Willow docs and get a good understanding. If it all looks good, I might be able to help out splitting these things out =].

[ Parent | on HN ]

Re:Willow Protocol by snvzz on Wednesday January 17, 2024 @08:31PM (#39036327)

iroh seems to have a couple of "killer tools" already, known as dumbpipe[0] and sendme[1].

Although I am concerned that while dumbpipe does mention cryptography, sendme's webpage makes no mention of it (?).

0. https://www.dumbpipe.dev/

1. https://iroh.computer/sendme

[ Parent | on HN ]

Re:Willow Protocol by rklaehn on Thursday January 18, 2024 @01:55AM (#39038800)

It's using the same transport. Basically sendme is like dumbpipe, but adds blake3 verified streaming from the iroh-bytes crate for content-addressed data transport.

[ Parent | on HN ]

Re:Willow Protocol by snvzz on Thursday January 18, 2024 @05:17AM (#39040054)

I imagined as much, but the website does still not mention encryption.

Which works against it. E2EE is a requirement today.

[ Parent | on HN ]

Re:Willow Protocol by rklaehn on Thursday January 18, 2024 @06:38AM (#39040577)

Thanks for letting us know. We will add a section about encryption.

These tiny tools are basically one week projects to show off the tech, but they try to be useful on it's own as well.

[ Parent | on HN ]

Re:Willow Protocol by nobodywillobsrv on Friday January 19, 2024 @07:09AM (#39054513)

Is iroh suitable for use as a collaborative decentralized function cache? Had a quick skim but can't quite see how to get at the get/put etc in the python api yet. Will spend more time later.

[ Parent | on HN ]