It was actually very elegant. There were 10 copies of PP state (20 in the 7600) and only 1 actual execution unit (2 in 7600). Hardware multi-threading in 1959! So there were 10 PP's executing PP overlay code (drivers) at 1/10 the instruction rate of the main CPU. Each PP had 4K of 12-bit words, which served for both PP code and I/O buffer space. The main memory was 60 bits wide (12*5) and the addresses were 18 bits, so the PP's had 18 bit accumulators for computing addresses.
Since PP's ran only trusted code, they were allowed to scribble anywhere in main memory that they wanted to. At the end of the I/O operation, the PP computed an address for the main CPU and that directly became the interrupt vector address. This meant that the CPU never had to deal with low level interrupts, only the much less frequent I/O operation completion interrupt at the end of a long operation.
(In a past life, I did system software at CDC, and CPU logic design at Sperry-Univac and Amdahl.)
Now I need to ask him about DMA architecture...
So regardless of MC's attractiveness from a systems standpoint, it was a nonstarter in an industry that was headed towards an ecology of clones and racing to the bottom in manufacturing costs. Many PC manufacturers ditched parity memory, too.
So the channel controller system has DMA; just not the peripheral.
> Minicomputers of the 1970s had low transistor counts and slow CPUs. So peripherals were usually put directly on the memory bus, with full access to memory.
That's "bus mastering" DMA. There is such a thing as "third party" DMA, which uses generic DMA controller, which is programmed to push data back and forth between memory and relatively dumb peripherals.
That approximates the channel concept.
PC's have had DMA controllers since 1980 something. The IBM PC had it in the form of the Intel 8237 chip. This is documented as having four "channels", wouldn't you know it.
Applications will typically specify transfers using virtual addresses, but DMA controllers will understand physical memory (usually, but maybe there are virtually mapped ones out there). Plus there are other issues like restricted ranges: DMA that cannot acces all physical memory but just a certain range. The API will have to convert ranges of virtual addresses to one or more physical ranges, which are then queued as one or more DMA operations and somehow deal with the problem that not just any old memory allocation in the application is DMA-able.
Applications talking to channel hardware directly using dedicated CPU instructions is only possible in some vendor-locked IBM mainframe world. It's not otherwise feasible simply for business/market/economic reasons having nothing to do with technical feasibility.
An operating system API is basically a set of extensions to the instruction set available in the application's virtual machine; it's no different from some dedicated I/O instructions, just perhaps less efficient (which may or may not matter).
Yup, this is what I’ve found on many embedded SoCs. It’s unfortunate. The fact that they need physical addresses makes them not worth using in many cases (mainly when trying to move user buffers around).
How does this not describe a PCIe bus master?
Frankly I don't understand any of this. High-performance I/O works by accessing main memory directly, just like it always has. The CPU then has to wait on main memory, just like it always has. Saying that the CPU is somehow the bottleneck seems to fall into the "not even wrong" area. There is no danger of I/O bandwidth approaching L1 or even L2 speeds anytime soon.
In the real world, if you need to deal with 400 Gb/s, you aren't going to send it to a single general-purpose CPU using any type of bus or channel. The CPU won't see it until another ASIC (or FPGA, I suppose) crunches it first.
Yes, that may impose a limit on the speed of an external network that a CPU can deal with, but that's the way it goes. MCA was never going to save us at this end of the Moore's Law curve.
So you need the hardware to divvy traffic up to multiple queues -- rings, really -- and a core for each ring. The packets just show up in memory, and the cores had just better keep up. If that looks to you like the same old POSIX architecture, I don't know what to say.
It was around 2010 that the network pipes got faster than the SAN file servers, and suddenly file output capacity had to be incorporated into network flow control.
Once you have two, and you want portable applications, using channel I/O instructions directly inline goes out the window; you need an API.
Porting code is easier when there are portable APIs. But that doesn't diminish from the quality of the abstraction.
From the paper:
> NVMe SSDs perform I/O faster than the OS can accept new I/O requests and notify their completion. Furthermore, the current POSIX AIO implementations in Linux are ugly and adhoc, and they have limited support from file systems. This has lead to new AIO interfaces that eliminate system calls and leverage polling. However, polling for completion is not suitable for large I/O transfers.
AWS Nitro SR-IOV I/O virtualization [1] uses Intel VT-d Posted Interrupts [2] to avoid polling, but this CPU feature is price-segmented to Xeon E5 and higher. For other CPUs, polling [3] is necessary to achieve high IOPS with NVME. Intel and AMD should consider making this feature available on all CPUs, to support NVME and NVDIMM.
[1] SR-IOV: https://www.snia.org/sites/default/files/RonEmerick_PCI_Expr... & https://www.twosixlabs.com/running-thousands-of-kvm-guests-o...
[2] IOMMU: https://www.linux-kvm.org/images/7/70/2012-forum-nakajima_ap...
[3] Polling: https://events.static.linuxfound.org/sites/events/files/slid... & https://www.snia.org/sites/default/files/SDC/2018/presentati...
The Nitro system does use SR-IOV, and it can take advantage of hardware virtualization features like VT-d posted interrupts to lower interrupt virtualization overhead. But VT-d posted interrupts isn't a factor in avoiding polling. I'd go further to say that the Two Six Labs blog post has some inaccuracies, so I wouldn't depend on the information it contains much.
Polling is often more efficient, and lower latency, than interrupt driven operation. If a CPU is mostly doing IO, polling for work can be a much better approach. Jens Axboe has been working on adding polling to the I/O stack for a while [1], and has recently added a new API for submitting I/O that is very promising [2] (25-33% performance improvements on I/O heavy workloads).
[1] https://lwn.net/Articles/705315/
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
I need these! Not for communication between the program and peripherals. I'd like to have these for communication between processes, threads, and software Actors. I'd like to be able to map such hardware channels to channels in Golang. What we have today are software channels implemented with, "heavy kernel mediation between the hardware and the user program," which means that one may be required to think a bit too much about how processes communicate with each other, and the performance implications and how these mechanisms can break down.
You can use VFIO, yes it needs an IOMMU but that basically what you ask for: hardware support - and yeah, I would like IOMMU to be generalized, also because it is dangerous to have some external ports without one, nowadays (even without thinking about efficient access from userspace). And because of that you can not delegate the core of the thing to "peripheral" anymore, basically you need IOMMU and once you have it, well what else do you need?
On the storage level, you can mmap on Optane on DDR4 without intermediate software copy I think. But that's merely an implementation detail anyway. When you think about the complexities of modern controllers and the diversity of options to implement them - and at the same time the most important latencies; well the only thing that matter is the end result, and obviously it can be better to not burn some CPU cycles in some cases (for some very specific workloads), but it is not trivial what to put in HW or SW to achieve that: RAM is already crazy slow, and the faster SSD is still even slower: so you will need to be smart in SW anyway to not be bitten by latencies. So in a sense; no! modern CPU are certainly not slow!
I also think you misrepresent the common modern interfaces availables for (even interoperable) SSDs; they are certainly not "emulat[ing] rotating disks".
I'd like that you give the following some thought to finish: 1/ HW backward compat is not a prime requirement anymore (because a/ smartphone dev model b/ even PC model has shifted enough so that it actually does not support complete backward compat anymore, mostly due to SoC integration). 2/ There is "sufficient" competition, where sufficient can be defined as effective to drive innovations because they can bring market advantages, and there are a non trivial number of competitors on the market (I'll mix a little but this is still relevant to what we are talking about: Intel, AMD, Arm, nVidia, Apple, Samsung, etc.) 3/ The designers working on those chips and systems are not stupid. What we have now is of course affected by technical history, but what we have now is already immensely different from what we had during the PS/2 era. Some of the historical impacts are possibly confined in less than a square millimeter on chips, and not only your PCI was not your ISA but your modern PCIe is also already quite different from your PCI... So the impact of peripheral vendors in the PS/2 era today for all of that: absolutely null, without a doubt.
Not sure where this exploit[1] stands now, but you may need a clean slate implementation that doesn't have to boot up with IOMMU disabled for legacy reasons.
[1] https://journal-bcs.springeropen.com/articles/10.1186/s13173...
What are some examples of these processors?
I would be interested in reading more about these mainframe processors architecture. Might you or anyone else have some links?
Closest to complex "IO Channel Program" on PCs was I2O which largely tanked despite leaving huge mark on SCSI controller design.
The peripheral never has access to main memory. There is no peripheral-controlled "direct memory access" (DMA). So it's possible to give control of a channel to a userland program without a memory security risk.
Minicomputers of the 1970s had low transistor counts and slow CPUs. So peripherals were usually put directly on the memory bus, with full access to memory. I/O operations were performed by storing into memory addresses, which caused bus transactions detected by the peripheral device. There were no CPU I/O instructions.
Microprocessors copied the minicomputer model. IBM's people knew this was a bad idea, and in the IBM PS/2, they introduced the "microchannel". Peripheral vendors, facing a new architecture that required more transistors, screamed. IBM backed down and went back to bus-oriented peripherals.
That model persists today, even though the few thousand transistors required for a channel controller are nothing today. Even though most modern CPUs have I/O channel like machinery, it's exposed to the program as registers the program stores into and memory accesses by the peripheral device.
So there's no standardization on how to talk to devices at the hardware level. Some CPUs have protection systems, an "I/O MMU", and there have been various channel-like interfaces, especially from Intel, but they have never caught on.
Instead, we mostly have heavy kernel mediation between the hardware and the user program. And way too many "drivers". This has become a problem with "solid state disk", which is really a random access memory device that doesn't write very fast. Mostly, it's used to emulate rotating disks.
Samsung makes a key/value store device which uses SSD-type memory devices but manages the key/value store itself. But you need a kernel between the device and the user program. You can't just open a channel to it and let the user program access it.