News for hackers. Stuff that matters.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by scurvy on Friday February 01, 2019 @04:14PM (#19058692)
You're doing something very wrong if that's what Ceph maxes out at. It's almost like you're trying to do this on a 1gig network.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by vvanders on Friday February 01, 2019 @04:38PM (#19058916)
Well it is a homelab, if you want to front me the cost of a 10Gbe switch be my guest ;).

Even then though I can get 125mb/s throughput from iperf, it's more just pointing out that Ceph has some pretty hefty bandwidth overhead. No knock against Ceph, it's a cool piece of tech and has its place if you've got the pipes to support it. The comment was more about how impressive ZFS is.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by scurvy on Friday February 01, 2019 @07:40PM (#19060167)
10gig stuff is pretty dirt cheap these days. I wouldn't mess around with 1gig and draw any conclusions about anything.

125MB/sec with iPerf is the max throughput of a 1gig link. Then again, iPerf doesn't do anything with the data. Ceph does a lot with the data once it's pulled off the network (mainly replication or erasure coding to other OSD's).

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by vvanders on Friday February 01, 2019 @08:47PM (#19060463)
Eeeeh. They're getting reasonable in price but I wouldn't call them dirt cheap. Since it's a homelab that lives in my house that rules out anything that sounds like a jet engine. If I match my other ubnt gear you're looking $400+ for a SFP+ capable switch.

I can hit 125MB/s on my ZFS migrations so it's not a unachievable number, just that Ceph had overhead(which is fine, it's doing something different than ZFS).

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by scurvy on Friday February 01, 2019 @09:45PM (#19060679)
You can find used 40gig switches that break out to 10gig ports at less than $4 per port. That's pretty cheap.

Even the older Arista and whitebox QCT stuff is less than $200-250 per switch. For a lab, I'd rather use something that once lived in a datacenter than something consumer-facing like Ubiquiti.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by vvanders on Friday February 01, 2019 @10:09PM (#19060777)
Well that's the beauty of a homelab, I can run whatever I want in it. Since it's all just for experimentation I don't see an issue running 1gbe. For what it's worth I did give a 3x bonded lcap link a try but since Ceph uses a single connection per node it doesn't scale across the aggregated link.

However at the end of the day if I don't want to keep the kid up at night I gotta run gear that doesn't sound like an F-18 at full throttle.