Actually, what Avi has demonstrated in this article is that XFS is the only filesystem that executes it mostly asynchronously.

Before we got started with the implementation of the I/O Scheduler (which we eventually wanted anyway for prioritization), I saw await time as reported by iostat as bad as 7s (truth be told, those weren't the best disks in the planet).

That was basically XFS sleeping during io_submit due to the problem I have briefly mentioned in this article, with the allocation groups.

If you limit the amount of requests the filesystem is consuming, then it is gone to the point that we started focused our attention in other areas. But it still has a couple of places in which it will resort to synchronous behavior.

No Linux filesystem can execute io_submit completely asynchronous.

I'm going to guess that your next step to chase further speeds is going direct to nvme just levering the block layer (per core multiq) skipping the filesystem entirely.

You're going to be right on that guess. It will take some time though.