Even then though I can get 125mb/s throughput from iperf, it's more just pointing out that Ceph has some pretty hefty bandwidth overhead. No knock against Ceph, it's a cool piece of tech and has its place if you've got the pipes to support it. The comment was more about how impressive ZFS is.
125MB/sec with iPerf is the max throughput of a 1gig link. Then again, iPerf doesn't do anything with the data. Ceph does a lot with the data once it's pulled off the network (mainly replication or erasure coding to other OSD's).
I can hit 125MB/s on my ZFS migrations so it's not a unachievable number, just that Ceph had overhead(which is fine, it's doing something different than ZFS).
Even the older Arista and whitebox QCT stuff is less than $200-250 per switch. For a lab, I'd rather use something that once lived in a datacenter than something consumer-facing like Ubiquiti.
However at the end of the day if I don't want to keep the kid up at night I gotta run gear that doesn't sound like an F-18 at full throttle.