News for hackers. Stuff that matters.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by deanCommie on Friday February 01, 2019 @10:33AM (#19054894)
"At Dropbox, Kafka clusters are managed by the Jetstream team, whose primary responsibility is to provide high quality Kafka services. "

And this is why AWS's "Managed Kafka" service has a place.

I get why Dropbox moved off S3 - storage is their core competency and they thought they could/should do better/cheaper. But I'm surprised they would be wasting valuable dev time with this kind of overhead.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @11:25AM (#19055467)
The AWS kafka product is a non-product. One would have to reproduce the same effort there as anywhere else.

60MB/s per broker

Took me about two hours worth of testing to reach that same conclusion. I like the Dropbox approach though, it is very thorough. One presumes that kafka is going to become a fundamental part of their infrastructure, in which case, this effort is a prudent investment.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by toomuchtodo on Friday February 01, 2019 @10:46AM (#19055000)
> But I'm surprised they would be wasting valuable dev time with this kind of overhead.

I'm not surprised at all. They have reached a scale where managed services are looked at with a critical eye, and are no longer the default.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by foobiekr on Friday February 01, 2019 @12:39PM (#19056319)
This. AWS is most valuable when the management overhead is a double digit percentage of your revenue. At some point the AWS margin itself is genuinely larger than DIY. That should be obvious as it has to be so unless AWS’s own costs are dramatically lower which is only possible if their value is smoke-and-mirrors.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @06:48PM (#19059902)
I don't think it's that hard to reach a scale where DIY makes sense. As was stated up above, the AWS kafka offering isn't complete. One could do something similar with a cloud formation template.

The ElasticSearch offering on the other hand is fairly complete. At my scale, it would cost me ~$15-20k/mo for ES in AWS. However, I DIY my own ES cluster (in k8s) which costs me something like $5k/mo. The delta is roughly $120-180k/yr.

An engineer costs me roughly ~$200k year with salary and benefits, so let's say the delta is .75 of an engineer. There are also risks to DIY that are hard to put an number on.

ES is definitely a PITA to run, but it's core enough to our infrastructure offering that dedicating .25-.5 of an engineer to it seems like a worthy trade-off to me. It is also a unique enough type of workload that it yields an interesting learning opportunity for my team.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by technion on Saturday February 02, 2019 @04:27AM (#19062051)
It also means you're in a position to pick up your stack and deploy it elsewhere if such a thing makes business sense in future. A lot of organisations are putting themselves in a position where that's not an option.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Sunday February 03, 2019 @12:52AM (#19068099)
If it's a given that cloud providers are aiming to get lock-in and increase margins, then it makes sense to do the necessary math to avoid that.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by toomuchtodo on Friday February 01, 2019 @08:01PM (#19060250)
I don’t disagree at all. Far too often, people wait too long to optimize away from a vendor’s offering. Their margin is your opex. Put that money back in your pocket.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by vaishaksuresh on Friday February 01, 2019 @12:30PM (#19056226)
As it stands today, AWS' managed kafka is very limited. Single region and no TLS is a non-starter for a lot of places.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by cbsmith on Friday February 01, 2019 @01:07PM (#19056652)
I'd use the Confluent managed Kafka service before I'd touch the AWS one.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @06:40PM (#19059869)
I've given it a test drive and highly approve. It is very polished.

Kinda spendy, though.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by cbsmith on Monday February 04, 2019 @05:39PM (#19081374)
Price is always negotiable. ;-)
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by Zecar on Friday February 01, 2019 @10:36AM (#19054912)
> storage is their core competency

Not so sure about that. I post in every one of these threads, but they deleted our honeymoon photos on my wife's account and then deleted my account several months later without notice. I wouldn't trust them with a swap file. Move fast and break things is a terrible mantra for a file storage company.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by pwaivers on Friday February 01, 2019 @10:41AM (#19054965)
Their core product is still storage, even if you had a bad experience. You may think Dropbox is a bad product, but OP's point still stands.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by xfitm3 on Friday February 01, 2019 @10:43AM (#19054976)
Isn't their core product the interface and software you use to access their storage?
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by linkmotif on Friday February 01, 2019 @10:47AM (#19055012)
Their core product is an interface, they resell cloud storage. Right?
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by gitgud on Friday February 01, 2019 @10:44AM (#19054977)
Can't say that I've ever had an issue with the reliability or service at Dropbox. Not sure they're known for moving fast either, as the product has changed very little in 10 years.