Even then though I can get 125mb/s throughput from iperf, it's more just pointing out that Ceph has some pretty hefty bandwidth overhead. No knock against Ceph, it's a cool piece of tech and has its place if you've got the pipes to support it. The comment was more about how impressive ZFS is.
125MB/sec with iPerf is the max throughput of a 1gig link. Then again, iPerf doesn't do anything with the data. Ceph does a lot with the data once it's pulled off the network (mainly replication or erasure coding to other OSD's).
I can hit 125MB/s on my ZFS migrations so it's not a unachievable number, just that Ceph had overhead(which is fine, it's doing something different than ZFS).
Even the older Arista and whitebox QCT stuff is less than $200-250 per switch. For a lab, I'd rather use something that once lived in a datacenter than something consumer-facing like Ubiquiti.
However at the end of the day if I don't want to keep the kid up at night I gotta run gear that doesn't sound like an F-18 at full throttle.
There's some pros to Ceph, mostly that you can add/remove drives at-will and the cluster will automatically rebalance. It's just very, very bandwidth hungry for cross-cluster communication.
It doesn't matter that much since it's for a homelab, I could migrate to some 10Gbe links between the cluster at some point and it'll get snappier. ZFS is just such a solid, battle tested piece of tech that it's hard to beat it unless you have a very different set of constraints.