Applications will typically specify transfers using virtual addresses, but DMA controllers will understand physical memory (usually, but maybe there are virtually mapped ones out there). Plus there are other issues like restricted ranges: DMA that cannot acces all physical memory but just a certain range. The API will have to convert ranges of virtual addresses to one or more physical ranges, which are then queued as one or more DMA operations and somehow deal with the problem that not just any old memory allocation in the application is DMA-able.
Applications talking to channel hardware directly using dedicated CPU instructions is only possible in some vendor-locked IBM mainframe world. It's not otherwise feasible simply for business/market/economic reasons having nothing to do with technical feasibility.
An operating system API is basically a set of extensions to the instruction set available in the application's virtual machine; it's no different from some dedicated I/O instructions, just perhaps less efficient (which may or may not matter).
Yup, this is what I’ve found on many embedded SoCs. It’s unfortunate. The fact that they need physical addresses makes them not worth using in many cases (mainly when trying to move user buffers around).
How does this not describe a PCIe bus master?
Frankly I don't understand any of this. High-performance I/O works by accessing main memory directly, just like it always has. The CPU then has to wait on main memory, just like it always has. Saying that the CPU is somehow the bottleneck seems to fall into the "not even wrong" area. There is no danger of I/O bandwidth approaching L1 or even L2 speeds anytime soon.
In the real world, if you need to deal with 400 Gb/s, you aren't going to send it to a single general-purpose CPU using any type of bus or channel. The CPU won't see it until another ASIC (or FPGA, I suppose) crunches it first.
Yes, that may impose a limit on the speed of an external network that a CPU can deal with, but that's the way it goes. MCA was never going to save us at this end of the Moore's Law curve.
So you need the hardware to divvy traffic up to multiple queues -- rings, really -- and a core for each ring. The packets just show up in memory, and the cores had just better keep up. If that looks to you like the same old POSIX architecture, I don't know what to say.
It was around 2010 that the network pipes got faster than the SAN file servers, and suddenly file output capacity had to be incorporated into network flow control.
Once you have two, and you want portable applications, using channel I/O instructions directly inline goes out the window; you need an API.
Porting code is easier when there are portable APIs. But that doesn't diminish from the quality of the abstraction.