There's a great blog post on this: https://blog.twitter.com/2015/building-distributedlog-twitte...

Which says " At design time we had concerns about Kafka’s I/O model and its lack of strong durability guarantees‐a non-starter for an application like a distributed transaction log[3]"

Seems reasonable, right?

Except "[3] Kafka addressed these durability concerns in version 0.8"

So they built a whole thing, because they didn't bother to ask or say "hey, if we help fix the durability, would that we welcome?" or even "do you guys have a plan and timeline to fix it?"

That's .... not great.

Now, maybe there are other reasons, but they aren't elucidated in this blog post :P.

Even then, my general view would be "did you approach the community and discuss your concerns or just dismiss them out of hand as infeasible", mainly because my experience is that if you do this, you often find they have exactly the same set of concerns/goals, and just need more resources to make it happen.

The desire to build shiny objects is very large. Outside of the paper plans of engineering teams, these things rarely end up up more shiny than what already exists or will be built by the time you are done.

Arguably there are benefits to developing in-house expertise, and no better way to develop expertise than to architect and build a solution end-to-end. Twitter now has several domain experts on staff who can continue maintaining DistributedLog and/or weigh its benefits against Kafka's and make more informed decisions going forward.

Not saying that they couldn't have worked more closely with Kafka's team in the first place, but, hey, now we have two Kafkaesque log services instead of just one. Seems like a win to me.

> Arguably there are benefits to developing in-house expertise, and no better way to develop expertise than to architect and build a solution end-to-end

There are also drawbacks to consider: Those experts might just decide to leave and then you have an in-house solution that you have basically no chance to find experts on. At least with an open source solution you may have an easier time.

> hey, now we have two Kafkaesque log services instead of just one. Seems like a win to me.

CantTellIfSerious.jpg

Or even "we are ready to pay 3 people for a year to build that, if we don't and give you the money instead what can you do ?".

The software industry has a real problem contributing to open source, the stuff, you know, allowing them to make money in the first place.

They did open source their solution. It sounds like you are arguing against parallel development. That happens all the time in open source.

Contributing to other open source projects is very different than to open source your project. NIH is one of the worst problem of open source : companies are spending a lot of money to dev new software instead of improving existing one.

Seriously this seems like a common pattern in open source: [big company] could just improve [X] but instead builds something from the ground up.

Sometimes you need to let talented engineers build things from the ground up, because it's good for them, makes them happy, and stops them from going to work somewhere else. Keeping people with the skills to solve these types of problems around and happy is also great for recruiting and for helping your less capable engineers learn and grow.

Sure, but duplicating Kafka? How many man-months did they put into building this, proving it out, and dealing with fallout from any bugs or production issues?

What's the point of retaining an engineer who's doing nothing for the business but re-inventing existing successful software?

"Sometimes you need to let talented engineers build things from the ground up, because it's good for them, makes them happy, and stops them from going to work somewhere else. "

actually, trying to let people do work you don't need done, in order to keep them happy, is a pretty rookie manager mistake.

If they aren't passionate about it, and you can't persuade them to do the things you need doing, they aren't the right person for the job.

That is always true, even if they were the right person in the past.

Your goal in that case should be to try find stuff the company needs done that they want to do, and push them to work on that. But if you find nothing, ...

Sometimes building something from the ground up allows you to make fundamental design decisions not available in the alternatives.

Or forks it and maintains an incompatible, non-contributable internal version based on a release from five years ago.