60MB/s per broker
Took me about two hours worth of testing to reach that same conclusion. I like the Dropbox approach though, it is very thorough. One presumes that kafka is going to become a fundamental part of their infrastructure, in which case, this effort is a prudent investment.
I'm not surprised at all. They have reached a scale where managed services are looked at with a critical eye, and are no longer the default.
The ElasticSearch offering on the other hand is fairly complete. At my scale, it would cost me ~$15-20k/mo for ES in AWS. However, I DIY my own ES cluster (in k8s) which costs me something like $5k/mo. The delta is roughly $120-180k/yr.
An engineer costs me roughly ~$200k year with salary and benefits, so let's say the delta is .75 of an engineer. There are also risks to DIY that are hard to put an number on.
ES is definitely a PITA to run, but it's core enough to our infrastructure offering that dedicating .25-.5 of an engineer to it seems like a worthy trade-off to me. It is also a unique enough type of workload that it yields an interesting learning opportunity for my team.
Kinda spendy, though.
Not so sure about that. I post in every one of these threads, but they deleted our honeymoon photos on my wife's account and then deleted my account several months later without notice. I wouldn't trust them with a swap file. Move fast and break things is a terrible mantra for a file storage company.
And this is why AWS's "Managed Kafka" service has a place.
I get why Dropbox moved off S3 - storage is their core competency and they thought they could/should do better/cheaper. But I'm surprised they would be wasting valuable dev time with this kind of overhead.