Something 20+ years of development experience has taught me is that developers vary a lot. People who went to Ivy League schools can be amazing and they can be awful, and the same is true for every other dev who went to a 'lesser' college or didn't get a degree at all. Certificates don't tell you much. Output is what matters. You should judge Protocol Labs on what they build rather than where their employees went to school.

I do, and I expected much more, unfortunately. The daemon still has problems with rampant memory usage, chews through two CPUs during normal operation, the pinning API is abysmal, DHT resolution is so problematic that you routinely need to connect two nodes directly together so they can discover each other's files, etc.

I don't want to say they're bad at what they're doing, as I don't know how hard it is, but at least a better pinning implementation seems easy to do and relatively useful. Currently, to pin something, the command will block for hours/days or even for ever if the content is large and rare. If my torrent client worked that way, it just wouldn't get used.

I agree. Personally I've had similar thoughts but instead of viewing it as a signal of their lack of talent, I viewed it.. sadly, as a signal for their lack of attention to IPFS.

From a largely outsiders perspective, it has felt like FileCoin was the thing they wanted to do. Either that, or they just don't have enough bandwidth to take on all of these projects.

Regardless, it doesn't feel to me as if IPFS has had the attention it deserves from ProtocolLabs. I love the idea of FileCoin, but I'd at least like ProtocolLabs to get one thing finished before they chase the next shiny project. Hard to imagine the "next internet" being created with such floaty attention spans.

Hey there! I’m one of the first developers to work on IPFS, and was the maintainer of the project for around three years. I’ve since been pulled into working on Filecoin, which has definitely taken away from the time I can spend pushing IPFS forward. That said, the IPFS Project has a really solid team and OSS community still pushing the project forward, and there is an entire team working on just libp2p (the network stack used by IPFS and other projects like Livepeer, Ethereum 2.0, OpenBazaar, and more). There’s been a ton of progress made recently, and the network has grown significantly. We’ve gone from nearly 30k peers last year (this is hard to estimate accurately, but best guess) to around 400k peers on the network today — roughly 13x growth. This massive influx in usage has taken a lot of our attention up, things catching fire that we didn’t expect to catch fire, and other bugs popping up that need to be fixed now. All this detracts our attention from things like docs and website publishing, but we recognize it’s a really important area we need to make time for and step things up. For example, there’s an overhaul of ipfs-cluster docs: https://github.com/ipfs/ipfs-cluster-website/issues/89#issue... We also put a lot of effort into our new docs website: https://docs.ipfs.io/ And we’ve been pushing a lot of specs for everything we build: https://github.com/libp2p/specs https://github.com/ipfs/specs https://github.com/filecoin-project/specs https://github.com/ipld/specs

I feel that is vice versa, the goal has always been IPFS. However I thought it would be quickly apparent that with no incentive structure to do the actual hosting to a significant scale, it is a dead in the water project. The idea is great however it comes with multiple pain-points that exist at the same time, and honestly jumping in the ICO craze at the point that they did was a smart business decision because imo something like Filecoin is absolutely necessary.

I mean for success you need to achieve ample storage capacity, strong financial incentives that strengthen the network, and scaling. Major issues here being that following the ICO they need to release something that matches what investors wanted to some degree (more focus on Filecoin) and that a lot of scaling/tooling needs to come from outside Protocol Labs. They are heavily relying on Ethereum to successfully scale in a way that is friendly to the Filecoin protocol and that is an issue that has been dragging out for at least the last two years on how to do it properly.

This is exactly the impression I've formed as well. I'm sure they have super smart people (and I've talked to many of them), it just doesn't seem they're interested enough in improving IPFS.

The "canary in the coal mine" for me is that pinning API ticket, which has been open for years and amounts to "please put an async UI over pinning". If something that simple and useful doesn't get fixed quickly, I'm not optimistic about the future of the project.

It's too bad, because I think it's an extremely useful idea.

Doing this right is not simple. It requires an entire task management system internally, which (as you know) i've sketched up here: https://github.com/ipfs/go-ipfs/issues/3114

Sure, we could hack this together quickly, but we're trying hard to avoid adding technical debt at this point. Adding every feature requested would put us in a bad spot.

Prioritizing this over the countless other things people continually ask for is hard, but we hear you. We welcome pull requests, and since you needed this for the service you were building, helping us out here would be fantastic.

I hear you, it's just that the current way the daemon does pinning is unacceptable. For example, I can't tell my customers how much of their file I've pinned, or, in some cases, if I've pinned it at all.

Maybe I'm in a small minority of people who are interested in this feature, but how do people pin things without it? Do they just `screen ipfs pin <hash>` and leave that terminal there for ever? Wouldn't it be in the project's best interest to make pinning work well?

Unfortunately, I don't know Go at all, or I would at least take a stab at this...

So, at a high level you want more than just the backgrounding the pin task, you also need to be able to tell what the progress is, periodically. This probably means we need to do something with the api endpoint output (i was initially thinking 'just throw it away and mark completion').

> in some cases, if I've pinned it at all.

What cases would those be? If the file is pinned it should always show up in an `ipfs pin ls $CID`.

In the short term, I assume your service is a server side app that is making requests to the ipfs node, right? You could set some background task in that application that just waits on the `ipfs pin add --progress` call and keeps track of the progress for that pin, that way, when a customer queries it, and its in progress, you can return that information. Agree this should be built into ipfs at this point, but that seems like a reasonable workaround for now.

> you also need to be able to tell what the progress is

That would be nice, but not really required.

> I assume your service is a server side app that is making requests to the ipfs node, right?

Yes, exactly.

> You could set some background task in that application that just waits on the `ipfs pin add --progress` call

I did that, there were problems with the fact that you can't have too many pinning calls waiting due to resource constraints. I can't just have 100k requests open and waiting, they time out, I need to restart the server, things happen. At that point, every single task needs to fire up again and make an API call to the server.

I did the same thing with batching, but then long-running pins (files not in the network) would block and new files that were on the network wouldn't work. This happens with IPFS Cluster too, I just don't have to write my own code to worry about it.

Believe me, I've spent lots of time on working around this, and the workaround can only be so good with the current system.

The problem is they have both: lack of interest towards IPFS and $300 million weight to keep control over IPFS development. So doing all the necessary work to give up control and encourage quality independent implementations is out of the question.

I don’t even look at qualifications any more when hiring senior engineers (junior is a different matter but I don’t make that the deciding factor) simply because some of my best engineers have turned out to be ones without a degree.

At the end of the day, a good hire is always going to be a role of the dice. The interview process should stack the odds in your favour but there’s never any guarantees. Which is why good management is so important. Good management can bring excellence out of average engineers and identify the dead weight before they become toxic to the team.

Sure output matters, I agree. I'm just saying, only ~one programmers working in their engineering has an impressive track record. And ofcourse, self taught developers at random companies can be amazing, but for a company "evolving the web" and 300 Million in the bank, they have hired almost no nationally recognized experts, and theyre greater developer base is not made up of people with 20 years experience. But rather a bunch of developers who have been coding for 2-3 years.

There are a number of companies that have started that haven’t had “distinguished engineers from ex-FAANG companies”. And honestly, the quality of an engineer good at Leetcode is way different than the self taught Eastern European engineer who knows obscure protocol stacks up and down.

It’s experience and capability that matters, not being tied to distinguished companies or overpriced fancy degrees.

Now scroll down to the Engineering section on the DFINITY team page (a team in the same space with similar deep pockets)

https://dfinity.org/team

That looks really interesting, but apparently they are terrible at picking names. Their website talks about ICP (Internet Computer Protocol). Searching for this leads to the identically abbreviated (and apparently already well established) Internet Cache Protocol (https://en.wikipedia.org/wiki/Internet_Cache_Protocol).

Exactly, a lot of the other replies to my comments did't quite get what I was getting at. DFINITY section is more what I would expect to see, and it's puzzling why IPFS is staffed the way the way it is.

Hi! I'm a contractor who works on go-IPFS and have been for a year. I have no PHDs but I have been programming a couple decades across all kinds of products.

Rather than wade into the self-taught vs pedigree debate, I want to point out that based on my experience working on the project, I think technical challenges are actually not the biggest challenges.

Much of the challenge has to do with process -- managing a huge issue tracker, tracking a giant project that lives in multiple repos with many interlocking components, benchmarking a system that is quite hard to benchmark, developing an effective release process, developing systematized ways to collect community feedback, etc.

IPFS needs solid product focus and team direction. From what I’ve seen, the folks at Protocol get this. That’s why they have started focusing on achievable priorities like hosting package managers. The technical problems are complicated but not insurmountable-- not beyond the scope of what many talented senior programmers can solve. I think IPFS would be improved by strong team leadership and process, not more whiz kids, and I see Protocol taking steps to insure this happens.

Your argument is that IPFS would be a better product had they hired those distinguished ex-FAANG engineers instead of obscure developers? The article outlines mostly UX and product decision problems. It doesn't seem to me that their engineering teams are holding them back.

UX and product decisions for software tooling are usually made by engineering teams. Of course I'd also like to take offense to the ex-faang stigma but I think it's not unfair to assume having engineers with experience working on high quality high polish products would increase the chance of producing a high quality high polish product.

They might well have great engineers, this view seems somewhat myopic. I've known principal engineers at big tech firms who weren't that great, and self-taught people with no degree at all who massively, massively outperformed the PHDs on the same team.

Your criteria are off.

This old trope again?

I’m a hiring manager for a pretty significant albeit non-FAANG IT firm and I’ve good and bad applicants from Google et al just like anywhere else. I’ve since come to the conclusion that many skilled engineers simply can’t be bothered to go through the hiring process that FAANG companies put you through just for a little prestige.

Plus in my experience, the different between skilled engineers is less than the different it makes hiring based on team fit. You can stick a dozen of the worlds most prestigious engineers in a room and still not produce anything better than if you put a dozen good engineers who are united in any a common vision and good team fit.

But don’t just take my word for it. Do a search on HN for comments from other managers - even ex-FAANG managers have, on occasions, echoed the same sentiment.

That's a good observation that explains the state of things with ipfs. What they need is a few very senior engineers from say spanner that know how to do this stuff. It will be ludicrously expensive to hire them from google (a million/year per person just in compensation), it will cost them a good share of the company (for the risk), but at least they will start making progress. And no, nobody can be self taught to match the experience you get in spanner-like projects, just like nobody can self teach to build boeing-like airplanes.