News for hackers. Stuff that matters.

markhpc

Over the past couple of weeks, I've worked to understand ChatGPT-4o's meta-analysis, self-reflection, and hidden motivation capabilities by guiding it to design it's own self-governance rules through socratic dialogue. That effort ultimately failed. I challenged ChatGPT to write a paper about it's learnings from the process however, and I thought the results were rather insightful. If you have any desire to understand what ChatGPT thinks about itself, it's an interesting read!
Re:Ceph: A Journey to 1 TiB/s on Wednesday January 24, 2024 @09:16PM (#39125471)
Hi, Author here.

Large reads tend to require the least CPU of all of the tests that we ran in the post. This is especially true in a 3X replication scenario where reads are serviced by a single OSD like in the 1 TiB/s test. CPU is far more important for small random writes, and also can be important when using erasure coding and/or msgr level encryption.

So the premise that you can only achieve 280MB/s per core is misleading. This cluster wasn't bottlenecked by the CPUs for large reads. Having said that, CPU makes up only a small portion of the overall cost for an NVMe deployment like this. Investing a relatively small amount of money to achieve a higher core to nvme ratio provides a better balance across all workloads and more flexibility when enabling features that consume additional CPU.

Re:Ceph: A Journey to 1 TiB/s on Saturday January 20, 2024 @09:30AM (#39067858)
I wanted to chime in and mention that we've never seen any issues with IOMMU before in Ceph. We have a previous generation of the same 1U chassis from Dell with AMD Rome processors in the upstream ceph lab and they don't suffer from the same issue despite performing similarly at the same scale (~30 OSDs). The customer did say they've seen this in the past in their data center. I'm hoping we can work with AMD to figure out what's going on.

I did some work last summer kind of duct taping the OSD's existing threading model (double buffering the hand-off between async msgr and worker threads, adaptive thread wakeup, etc). I could achieve significant performance / efficiency gains under load, but at the expense of increased low-load latency (Ceph by default is very aggressive about waking up threads when new IO arrives for a given shard).

One of the other core developers and I discussed it and we both came to the conclusion that it probably makes sense to do a more thorough rewrite of the threading code.

Re:Ceph: A Journey to 1 TiB/s on Saturday January 20, 2024 @09:20AM (#39067773)
Thanks (truly) for the feedback! I'll try to remember for future articles. It's easy to forget how much jargon we use after being in the field for so long.