eivanov89

But this is not

You are right :)

Thank you, that's actually was my actual attitude: found interesting things -> used AI to polish notes and quick draft -> "Hey folks, here is a cool thing I've found".

Thanks for sharing. I didn't expect my slavic accent to be so strong to be visible in text and even pass through AI edits :)

But when process dies, might it hope to become init on next reboot?

Fair enough.

Note, that this also includes retaining process memory related to the I/O. In my experiment I used regular 4 KiB pages, which is smaller or equal to the I/O block size. However, in case of huge pages only a small fraction can be I/O related. And I suppose, that this small part would pin the whole page, which might have some sensitive information. Still protected, however, but who knows.

That's a good approach. My sin is that if I like the sentence, I keep and don't rewrite. Because if I see it is good, I don't feel I should. And probably I don't see that "prose" is bad each time it is indeed bad.

Frankly speaking, you are the second (with this thread, the third) one, who says that. So, it looks that I really should write myself again with just grammar checks, not the style.

Thank you very much for a very nice annotation/summary and sharing your thoughts.

Jens Axboe, the io_uring author, promissed to address this on X next week: https://x.com/axboe/status/2095856971912028536?s=20 Stay tuned.

I'm the author and I'm not native speaker, so please bear that in mind. However, I admit I've used AI for grammar and style. IMHO, the post is not fiction, I just wanted to communicate the ideas as clearly and effectively as possible.

Hi folks, I’m the author. I submitted this yesterday under a different title. Giving it one more try. Happy to discuss the post and Linux async I/O.

Hi folks, I’m the author. Happy to discuss the post and async I/O in Linux

Thank you so much, I'm happy to hear that the article has been useful for you.

Hi folks, I’m the author. Happy to discuss the post, C++20 coroutines, or the TPC-C implementation here.

Hi folks, I’m the author. Happy to discuss the post, C++20 coroutines, or the TPC-C implementation here.

Dear friends, I'm the author of this implementation. Happy to discuss details.

> Provocative thought in that case would then be - in what % are io_uring improvements (over libaio) undermined by the device behavior (firmware) in mixed workloads. That % could range from noticeably to almost nothing so it might very well affect the experiment conclusion.

That’s absolutely fair. Also, it would be useful to test across different devices, since their behavior can vary significantly, especially when preconditioned or under corner-case workloads.

In our case, we focused on scenarios typical for YDB deployments, so we didn’t extend the study further. That said, we believe the observed trends are fairly general.

> For example, if one is posing the question if switching to io_uring is worth it, I could definitely see different outcomes of that experiment in mixed workloads per observations that you described.

I agree that for mixed workloads the outcome may differ. However, for us the primary concern in the AIO vs io_uring comparison is syscall behavior.

It is critical that submission does not block unpredictably. Even without polling, io_uring shows consistently better latency across the full range of iodepths. If device latency dominates (as in your scenario), the relative benefit may shrink, but a faster submission path still helps drive higher effective queue depth and utilize the device better.

> This is also interesting. May I ask why is that the case? Are you able to saturate the NVMe disk just with a single thread? I assume not but you may be using some particular workloads and/or avoiding kernel that makes this possible.

The component we are working on is designed for write-intensive workloads. Due to DWPD constraints, we intentionally limit sustained write throughput to what the device can safely handle over its lifetime. In practice, this is often on the order of ~200–300 MB/s, which a single thread can easily saturate.

At the same time, we care a lot about burst behavior. With AIO, we observed poor predictability: total latency depends heavily on how requests are submitted (especially with batching), and syscall time can grow proportionally to batch size * event count.

io_uring largely eliminates this issue by decoupling submission from syscalls and providing a much more stable submission path. Additionally, for bursty workloads we can use SQPOLL + IOPOLL to further reduce latency in specialized setups.

When IOMMU is not enabled, any PCIe device capable of DMA could access arbitrary physical memory. It allows to read any sensitive data, modifying memory and fully compromising the system without CPU involvement.

There are many DMA-based attacks described in the literature. Even with IOMMU, some attacks are still possible due to misconfiguration or incomplete isolation. For example: https://www.repository.cam.ac.uk/items/13dcaac4-5a3d-4f67-82...

In our case, we didn’t dive deeply into the security aspects. Our typical deployment assumes a trusted environment where YDB runs on dedicated hardware, so performance considerations tend to dominate.

We compared IOMMU fully disabled vs enabled. When it is enabled, I expect it to be lazy (should be the default for IOMMU). Note, that we recommend to use passthrough to completely bypass translation for most devices independent on strict/lazy mode.

There are a couple of challenges with mixed read+write workloads on NVMe.

In practice, read latency tends to degrade over time under mixed load. We observe this even across relatively short consecutive runs. To get meaningful results, you need to first drive the device into a steady state. In our case, however, we were primarily interested in software overhead rather than device behavior.

For a cleaner comparison, it would probably make sense to use something like an in-memory block device (e.g., ublk), but we didn’t dig into it.

As for profiling: we didn’t run perf, so the following is my educated guess:

1. With libaio, control structures are copied as part of submission/completion. io_uring avoids some of this overhead via shared rings and pre-registered resources. 2. In our experience (in YDB), AIO syscall latency tends to be less predictable, even when well-tuned. 3. Although we report throughput, the setup is effectively latency-bound (single fio job). With more concurrency, libaio might catch up.

We intentionally used a single job because we typically aim for one thread per disk (two at most if polling enabled). In our setup (usually 6 disks), increasing concurrency per device is not desirable.

That's super hot. Especially the update with the 37M IOPS reference. Might be very useful for my next tasks related to a setup with 6 NVMe disks: 1. Get all disks saturated through the network (including RDMA usage). 2. Play with io_uring to share a polling thread. Currently, no luck: if I share kernel poller between two devices then improvement is just +30% (at a cost of 1 core). Considering alternative schemes now.

BTW, the whole situation with IRQ accounting disabled reminds me the -fomit-frame-pointer case. For a long time there was no practical performance reason, but the option had been used... Making slower and harder to build stacks both for perf analyses and for stack unwinding in languages like C++.

After careful reading I'm surprised how small IRQ squares build up 30%. Should search for interrupts when I inspect our flamegraphs next time.