Re:Finding Kafka’s throughput limit in Dropbox infrastructure
by technion on Saturday February 02, 2019 @04:27AM
(#19062051)
It also means you're in a position to pick up your stack and deploy it elsewhere if such a thing makes business sense in future. A lot of organisations are putting themselves in a position where that's not an option.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure
by halbritt on Sunday February 03, 2019 @12:52AM
(#19068099)
If it's a given that cloud providers are aiming to get lock-in and increase margins, then it makes sense to do the necessary math to avoid that.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure
by toomuchtodo on Friday February 01, 2019 @08:01PM
(#19060250)
I don’t disagree at all. Far too often, people wait too long to optimize away from a vendor’s offering. Their margin is your opex. Put that money back in your pocket.
The ElasticSearch offering on the other hand is fairly complete. At my scale, it would cost me ~$15-20k/mo for ES in AWS. However, I DIY my own ES cluster (in k8s) which costs me something like $5k/mo. The delta is roughly $120-180k/yr.
An engineer costs me roughly ~$200k year with salary and benefits, so let's say the delta is .75 of an engineer. There are also risks to DIY that are hard to put an number on.
ES is definitely a PITA to run, but it's core enough to our infrastructure offering that dedicating .25-.5 of an engineer to it seems like a worthy trade-off to me. It is also a unique enough type of workload that it yields an interesting learning opportunity for my team.