News for hackers. Stuff that matters.
Re:Amino – The Public IPFS DHT Is Getting a Facelift by stavros on Tuesday October 03, 2023 @07:41AM (#37750529)
Is IPFS working these days? I was very excited about it eight years ago, to the point where I made one of the first IPFS pinning services, but lost all my interest. IPFS is a great idea, but the implementation basically doesn't work, and it certainly doesn't work to the point where people can be running the node locally.

It used to have tons of problems discovering content from other nodes on the network unless it was directly connected to them, and it broke often. It also didn't seem like Protocol Labs worked on any of these problems at all, focusing on launching a cryptocurrency instead.

Has it changed now?

Re:Amino – The Public IPFS DHT Is Getting a Facelift by Borg3 on Tuesday October 03, 2023 @07:46AM (#37750573)
Well said. I too was looking at IPFS few times, but hardly could find a place to use it. Im a big fan of distributed storage or self hosting, but IPFS is far too static for anything usefull, except maybe archives of import static blobs.
Re:Amino – The Public IPFS DHT Is Getting a Facelift by diggan on Tuesday October 03, 2023 @07:53AM (#37750637)
What exactly are you unable to build with IPFS that requires it to be even more dynamic than what it is? I've found it flexible enough for most use cases I've had in mind, as long as you're flexible on how they architecture should look.
Re:Amino – The Public IPFS DHT Is Getting a Facelift by Borg3 on Tuesday October 03, 2023 @01:18PM (#37754804)
Thats the point. I cant kinda wrap myself into it. I cant see it anything more that distributed object storage. So, calling it a future web is kinda exaggeration imo. Not sure. I host very simple web pages myself, and yet they are pretty dynamic on output.

Can you point me to some interesting projects that utilize IPFS?

Re:Amino – The Public IPFS DHT Is Getting a Facelift by 2color on Wednesday October 04, 2023 @10:11AM (#37765459)
See https://awesome.ipfs.tech/ and https://ecosystem.ipfs.tech/ for some project using it.
Re:Amino – The Public IPFS DHT Is Getting a Facelift by Borg3 on Wednesday October 04, 2023 @02:20PM (#37769411)
Thx. Seems im too old fashioned or sth.. Most projects are out of scope. Some are interesting, and some are weird, like IPFS chat or Push to talk.

I think I will stick to my good old web for now :)

Re:Amino – The Public IPFS DHT Is Getting a Facelift by diggan on Tuesday October 03, 2023 @07:51AM (#37750616)
Well, I guess it depends on your use-case. As a general and public discovery / providing / downloading network, it's been kind of overloaded for the last years, and it finally seems like Protocol Labs is putting some efforts to solving some of the most biting issues.

In this case, it's about the process of adding content to the network. It was neigh impossible to add large directories/files as your connections got overloaded with provide messages. This seems to batch things up and parallelize better, so should at least make it easier to add content and subsequently find it for the peers who want to.

But, the implementation still works very well when you're doing your own networks, which I think is a much better use of the protocol anyways. So when building a application with IPFS, you're using your own network only composed of nodes that are actually relevant to your application, instead of connecting to the public DHT.

Unless your scale is really big, it'll work a lot better than using the already huge public DHT.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by londons_explore on Tuesday October 03, 2023 @08:29AM (#37750961)
I think it would be better to fix the public network than to split the network into millions of local networks...

The public design today still allows actual file transfers to be local over your local network - it is only metadata that goes over the public internet.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by diggan on Tuesday October 03, 2023 @08:35AM (#37751020)
Sure, ideally that'd be the case. But today that kind of isn't feasible as discovering and providing content is slow, but hopefully it gets better in the future with changes like this and more.

But what I wrote is how you can solve the issue today.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by 9dev on Tuesday October 03, 2023 @08:21AM (#37750893)
I hope the irony of a protocol called „Interplanetary file system“ being more suitable for local usage isn’t lost on people :)
Re:Amino – The Public IPFS DHT Is Getting a Facelift by diggan on Tuesday October 03, 2023 @08:23AM (#37750911)
I hope you don't believe I wrote something about that IPFS is only suitable for local usage. You can have your own remote network, spanning whatever nodes/computers/servers you want, remote or otherwise :)
Re:Amino – The Public IPFS DHT Is Getting a Facelift by _factor on Tuesday October 03, 2023 @03:50PM (#37756669)
Funny enough. Interplanetary refers to the ability to know a file created on earth would still hash to the same value if an alien created it on Alpha Centauri. You could both work independently and combine your databases and the distance wouldn’t matter.

That’s seems pretty local to me.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by stavros on Tuesday October 03, 2023 @08:56AM (#37751233)
That's kind of a shame, the appeal of a public, peer-to-peer, content-addressable network was very high to me personally, because of its relative uniqueness. For a personal network, IPFS becomes just another technology I could deploy, out of many options.
Re:Amino – The Public IPFS DHT Is Getting a Facelift by diggan on Tuesday October 03, 2023 @09:23AM (#37751515)
Yeah, definitely I feel the same about the public parts, but I still find the public-but-private network part of it useful in itself. Both have use cases, but currently the public one just isn't feasible to use :/
Re:Amino – The Public IPFS DHT Is Getting a Facelift by theK on Tuesday October 03, 2023 @07:56AM (#37750659)
I also was very active in the early IPFS days. I think two points really atributted to your experience

1. Success: IPFS got tons of usage early on so scaling the software (which back then was mostly a prototype) was challenging, especially with Bennett's BDFL initial stance.

2. The need to codify an incentives market which then lead to the creation of filecoin took a lot of effort and setting up a trustworthy org around that one and ipfs got even more challenging

So Yes, working with IPFS was not plain sailing (and still isn't) but it seems that by now the two projects have been set up to start iterating again and I see a lot of great work happening on both fronts so it looks lika a promising future here.

Source: I have worked and am still working with both IPFS and Filecoin as part of my business

Re:Amino – The Public IPFS DHT Is Getting a Facelift by noman-land on Tuesday October 03, 2023 @10:17AM (#37752307)
Can you say a bit about using it for your business? Curious how people are using IPFS in the real world.
Re:Amino – The Public IPFS DHT Is Getting a Facelift by theK on Tuesday October 03, 2023 @12:27PM (#37754081)
Apart from the obvious web3 stuff there are some interesting usecases for classical enterprise software.

Caveat, my experience is limited to clusters below 100 nodes.

You can run your private IPFS DHT if you want and the protocol has some cool load shedding features. There is also IPFS cluster which adds data availability features to your IPFS deployments. When we tried private IPFS networks the cluster project was relatively early and we found that managing DA in the application that owns the data ended up being more straight forward so we didn't use that but I've heard good things lately.

If you don't need to enforce access management across your internal data calls it can be an interesting alternative to stuff like minio or ceph. YMMV but my experience was that, running a private IPFS cluster can be easier and much more flexible than running a similarly scaled minio or ceph deployment. It can be seen as a shift left like DevOps is. Instead of having each internal app on your project depend on a Data team that manages Ceph or another data storage and access technology you run (most of) it yourself.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by zubairq on Tuesday October 03, 2023 @02:40PM (#37755854)
I’m actually using it for sharing enterprise components so that a document saved in one company can be sent to another company and the document loads content from IPFS. All this without needing a central server
Re:Amino – The Public IPFS DHT Is Getting a Facelift by chriswarbo on Tuesday October 03, 2023 @11:08AM (#37752932)
> 2. The need to codify an incentives market which then lead to the creation of filecoin took a lot of effort and setting up a trustworthy org around that one and ipfs got even more challenging

I see this as a non-goal: HTTP is doing fine without an "incentives market", and that's the sort of core layer IPFS is suited for. When I switch off my HTTP servers, there's no expectation that the resources they're hosting remain accessible; and the same is true for IPFS. The advantage of IPFS is that it allows resources to remain accessible, e.g. if someone else cares enough to host it too, or if I happen to have copies buried on some old boxen (without having to coordinate some load-balanced shenanigans up-front).

For example, we can avoid "leftpad" fiascos if software companies could host their own dependencies (as in, contribute to ensuring their canonical URLs resolve; rather than current practice of re-hosting copies at myriad private URLs, or routing their network through caching proxies).

Good luck to other projects which want to work on such a thing (Filecoin, etc.), but it's mostly orthogonal to IPFS itself.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by theK on Tuesday October 03, 2023 @12:06PM (#37753746)
> The advantage of IPFS is that it allows resources to remain accessible, e.g. if someone else cares enough to ho...

I think there is a key point here. This exactly is the reason why incentives markets built around IPFS are so relevant. They democratize the act of saying "I care about that data". Nowadays you need to be a poweruser to express this but with the world's content being more and more digital media this isn't the realm of the poweruser any more.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by chriswarbo on Tuesday October 03, 2023 @05:28PM (#37757807)
> I think there is a key point here. This exactly is the reason why incentives markets built around IPFS are so relevant.

I fundamentally disagree. If someone wants to make filesharing easier, go ahead (personally I think dragging files into a "Shared" folder, like eDonkey/Kazaa/Limewire/etc. would be easier for non-"powerusers" than minting and securing a crypto wallet and trying to convince others of its value in exchange for storage space, but whatever).

To me, such projects seem about as relevant for IPFS as, say, Geocities is for HTTP.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by theK on Tuesday October 03, 2023 @05:58PM (#37758171)
I'm having difficulty following this train of thought. How is any of this related to consumer level users wanting to keep IPFS data alive?
Re:Amino – The Public IPFS DHT Is Getting a Facelift by chriswarbo on Wednesday October 04, 2023 @08:52AM (#37764358)
> How is any of this related to consumer level users wanting to keep IPFS data alive?

I'm saying that "consumer level users wanting to keep... data alive" is unrelated to IPFS.

If someone wants to work in that space, they can do so in separate projects (there are already many). Such projects could make use of IPFS; but they could also use HTTP, Bittorrent, GNUNet, Dat/Hyperdrive, Freenet, ED2K, FTP, or whatever seems appropriate (ideally, all of the above and more).

Presumably such projects could encode their text using UTF-8, to 'democratise saying things outside of the Anglosphere'; but that shouldn't de-rail the Unicode consortium into a multi-year crypto pivot.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by theK on Wednesday October 04, 2023 @01:47PM (#37768928)
That's all fine but the fact of the matter is that a lot of web3 content is built on IPFS and not on all those other technologies (partially because IPFS is truly permissionless), so the "just use another tech" argument is quite moot/academic
Re:Amino – The Public IPFS DHT Is Getting a Facelift by chriswarbo on Thursday October 05, 2023 @01:23PM (#37781335)
> the "just use another tech" argument is quite moot/academic

What "argument"? I said that filesharing/storage-incentivising projects can build on whatever they like; and that they have been doing, for decades.

My "argument" is that the IPFS project/protocol is not such a project. It is a network protocol (or suite thereof; for peer discovery, data transfer, hash representations, etc.).

> the fact of the matter is that a lot of web3 content is built on IPFS and not on all those other technologies

So what? Hopefully those projects chose to use IPFS due to its technical features, its stated goals, etc. How's that working out for them?

Note that, in particular, those technical features and stated goals do not include getting other people to host your shit. If some "web3" projects have decided to build their crypto bullshit on top of such a fundamental misunderstanding, then that's hilarious! More realistically, I'm guessing they know full-well that's not how any of this works; and are just keeping the illusion going until their inevitable rug-pull.

Either way, that's not IPFS's problem (which, again, is a (suite of) network protocols).

Re:Amino – The Public IPFS DHT Is Getting a Facelift by vineyardmike on Tuesday October 03, 2023 @07:11PM (#37758823)
Incentive markets aren’t “democratization”. Something like BitTorrent where people freely choose to contribute simply to help others is democratic.

An incentive market basically makes it a business.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by theK on Wednesday October 04, 2023 @04:11AM (#37762280)
What's not democratic in having the power to use and operate businesses?

I think this is the ideal way. Instead of limiting "democracy" to the power users you also include consumers this way.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by ianburrell on Tuesday October 03, 2023 @12:50PM (#37754400)
I bet IPFS could have gotten by with mirroring system. Sites publishes list of files that they want mirrored, and supporters could copy and host the files. I think IPFS could be more useful for software, and mirroring or caches would work well.

Also, IPFS was supposed to have clients host the file like Bittorrent does. My impression is that most users don't host, they just download. It is quite possible that adding money to system reduces people's desire to contribute. I also got the impression that flaws in client make it harder to host.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by KirillPanov on Tuesday October 03, 2023 @07:58AM (#37750681)
IPFS is propped up by AI companies.

When it becomes clear that their models were trained on library genesis, they are betting that "but our web crawler stumbled into it through cloudflare's gateway" will be a good enough excuse to keep them out of prison.

This is basically the only thing IPFS offers that Bittorrent doesn't.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by j_maffe on Tuesday October 03, 2023 @08:18AM (#37750871)
> This is basically the only thing IPFS offers that Bittorrent doesn't.

Now that's simply just untrue. The main difference from Bittorrent is that it relies on Content ID, not torrent links. IPFS is used by many organizations and individuals more than just AI companies.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by yieldcrv on Tuesday October 03, 2023 @08:32AM (#37750994)
It works great for my use case

Since you havent looked, people are using Protocol Lab’s crypto version of IPFS to pin on IPFS

Filecoin+IPFS is far more free than any of the IPFS SaaS pinning services

and it has decent replication too

I serve over CDNs, of which there are many, and they cache well enough.

I use it to stay on Vercel and Netlify’s free tiers for my static assets, so my sites can have huge spikes in traffic but my static assets are not loaded from them.

Its free on free, big use case for exploratory projects

https://web3.storage does that filecoin+ipfs pinning

Re:Amino – The Public IPFS DHT Is Getting a Facelift by yiannisbot on Tuesday October 03, 2023 @09:12AM (#37751379)
We've been doing quite extensive measurements over several parts of the architecture, which you can find here: https://probelab.io/

Still lots to optimise, but wouldn't say it's unusable. In fact, performance is pretty good for a decentralised P2P network.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by simonw on Tuesday October 03, 2023 @12:26PM (#37754055)
I mostly lost interest in it when I learned that it's possible for a file published to IPFS to simply blink out of existence one day in the future (if no one is left pinning it).

At that point I'd rather stick something in an S3 bucket and pay for it myself.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by stavros on Tuesday October 03, 2023 @12:29PM (#37754108)
I didn't mind that too much, because IPFS is strictly better than S3 in that regard. IPFS isn't meant to make it so that you don't need to host your own files, but rather that I can seamlessly host your files too.

In that regard, it's much more available than any HTTP server.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by detaro on Tuesday October 03, 2023 @01:43PM (#37755148)
a file on S3 will simply blink out of existence too one day if you stop paying for one specific someone to "pin" it, so I don't think this point makes that much sense without further context?
Re:Amino – The Public IPFS DHT Is Getting a Facelift by simonw on Tuesday October 03, 2023 @02:11PM (#37755509)
I care about making things available to other people. My initial attraction to IPFS was that it looked like it could help me do that - but then I learned that just publishing a file to IPFS doesn't permanently solve that problem for me.

It solves caching/distribution, but it doesn't solve make-this-thing-available.

I've been using and paying for S3 for this purpose for 15+ years. I was hoping IPFS could offer a better alternative, but I don't think it does.

I might consider IPFS in the future if I need to distribute a prohibitively large file and my target audience are the kind of people who can access IPFS.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by Karrot_Kream on Tuesday October 03, 2023 @04:07PM (#37756867)
The Cloudflare gateway works pretty well for this honestly. You can send an HTTP request yourself to the Cloudflare gateway after pinning a file and Cloudflare will generally have it available.
Re:Amino – The Public IPFS DHT Is Getting a Facelift by kkielhofner on Tuesday October 03, 2023 @12:27PM (#37754078)
I have a similar take with slightly more recent experience.

When it came down to it the resource requirements for an IPFS node were pretty insane relative to the "value" provided and by many takes it still basically didn't "work".

I understand it's not the same thing at all but in the days of running a web server on nearly anything than can handle many thousands of requests/sec an IPFS node running on the beefiest hardware we could throw at it ate tremendous amounts of system resources and bandwidth for double digit requests per second, and even then it would frequently time out and/or get into various unrecoverable states necessitating a service restart. We had to run a cluster of them and watch IPFS nearly melt the hardware...

We tried every IPFS implementation available and ended up having to use the "least worst" while also adding a lot of instrumentation, health checks, etc around it just to keep it up and running in some kind of consistent, usable fashion.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by aftbit on Tuesday October 03, 2023 @02:39PM (#37755845)
I briefly ran an IPFS node, I believe working towards the same project that you are discussing. It ate my home network: drove my packet loss into the 10% range and somehow convinced my core switch (a Brocade ICX6610) to send all traffic to every port. When I saw every port on my upstairs switch blinking like crazy and tcpdump showed traffic intended for a downstairs server arriving at my upstairs workstation, I pulled the plug and told free he was on his own.
Re:Amino – The Public IPFS DHT Is Getting a Facelift by kkielhofner on Tuesday October 03, 2023 @04:27PM (#37757100)
This was for an internal project, but yeah your experience sounds about right.
Re:Amino – The Public IPFS DHT Is Getting a Facelift by omginternets on Tuesday October 03, 2023 @12:40PM (#37754237)
I'm quietly using it in a side-project of mine that is intended to provide a cloud-esque environment to a permissioned p2p compute cluster. In my case, it's basically providing S3-like functionality, which works rather nicely in a datacenter environment.
Re:Amino – The Public IPFS DHT Is Getting a Facelift by evbogue on Tuesday October 03, 2023 @12:49PM (#37754370)
I don't mind pinning, but back in the day I was having issues using IPFS to transfer small files between devices. I admit I haven't investigated to see if the problems were ever resolved.

I've been watching https://github.com/ipfs/helia which is going to replace https://github.com/ipfs/js-ipfs and hoping they can get an IPFS node working in the browser.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by b_fiive on Tuesday October 03, 2023 @02:34PM (#37755780)
Depends on how you define working :). I'm a 6+ year vet of the IPFS ecosystem, we work on iroh these days, which I think addresses many core issues with the protocol design: https://iroh.computer

The biggest challenges still unaddressed are twofold imho: 1. The network is very forgetful. Stuff you added 24 hours ago is likely gone unless you've taken specific steps to keep it up. This is hard because all CIDs in IPFS have equal weight, which makes it very hard to cache intelligently. 2. The implicit promise that IPFS will resolve _any_ of the 86-100k new CIDs it sees daily in "normal internet" times, (sub-second TTFB). This doesn't work in practice, because mapping content addresses to location-based providers who are under high churn is, well, very hard.

Both of these problems are "content routing" problems, which is the core of "get me stuff from this hash, I don't care where" interface IPFS offers. It's hard. With iroh we just don't make that promise at all right now.

Re:Amino – The Public IPFS DHT Is Getting a Facelift by stavros on Tuesday October 03, 2023 @03:22PM (#37756318)
I hope you succeed!
Re:Amino – The Public IPFS DHT Is Getting a Facelift by anacrolix on Wednesday October 04, 2023 @04:41PM (#37771434)
IPFS has ruined the public perception of what a content addressable network could be.

Now when you mention CAD, people think IPFS and freak out about bad performance and flakiness. It's a real shame because we already have a fantastically reliable CAD and DHT in BitTorrent, and it's trivial to build on top of that to create excellent experiences.