News for hackers. Stuff that matters.
Re:BTFS (BitTorrent Filesystem) by infogulch on Monday April 15, 2024 @10:29PM (#40047823)
It's not an overlay provider itself, but uber/kraken is a "P2P Docker registry capable of distributing TBs of data in seconds". It uses the bittorrent protocol to deliver docker images to large clusters.

https://github.com/uber/kraken

Re:BTFS (BitTorrent Filesystem) by XorNot on Monday April 15, 2024 @11:47PM (#40048173)
The problem with being a docker registry is that you're still having to double-dip: distribute to the registry, then docker pull.

But you shouldn't need to: you should be able to do the same thing with a docker graph driver, so there is no registry - even daemon should perceive the local registry as "already available", even though in reality it's going to just download the parts it needs as it overlay mounts the image layers.

Which would actually potentially save a ton of bandwidth, since the stuff in an image is usually quite different to the stuff any given application needs (i.e. I usually base off Ubuntu, but if I'm only throwing a Go binary in there plus wanting debugging tools maybe available, then in most executions the actual image pulled to the local disk would be very small).

Re:BTFS (BitTorrent Filesystem) by infogulch on Tuesday April 16, 2024 @05:35AM (#40050041)
Re:BTFS (BitTorrent Filesystem) by apichat on Tuesday April 16, 2024 @03:59AM (#40049472)
https://github.com/uber/kraken?tab=readme-ov-file#comparison...

"Kraken was initially built with a BitTorrent driver, however, we ended up implementing our P2P driver based on BitTorrent protocol to allow for tighter integration with storage solutions and more control over performance optimizations.

Kraken's problem space is slightly different than what BitTorrent was designed for. Kraken's goal is to reduce global max download time and communication overhead in a stable environment, while BitTorrent was designed for an unpredictable and adversarial environment, so it needs to preserve more copies of scarce data and defend against malicious or bad behaving peers.

Despite the differences, we re-examine Kraken's protocol from time to time, and if it's feasible, we hope to make it compatible with BitTorrent again."

Re:BTFS (BitTorrent Filesystem) by anacrolix on Tuesday April 16, 2024 @09:16AM (#40051496)
That's exactly the problem my BitTorrent client solves. https://github.com/anacrolix/torrent. It's been around since 2013.
Re:BTFS (BitTorrent Filesystem) by phillebaba on Tuesday April 16, 2024 @04:14AM (#40049544)
Kraken is sadly a dead project, with little work being done. For example support for Containerd is non-existent or just not documented.

I created Spegel to fill the gap but focus on the P2P registry component without the overhead of running a stateful application. https://github.com/spegel-org/spegel