You can do some pretty cool things from a HW designer's perspective inside the accelerator, and in the main application. Since the accelerator is cache-coherent, and able to map the same virtual addresses as a given process (and attach to multiple processes' address spaces) the device can do "simple" things like follow pointers, which used to require building a command / data packet, DMA'ing it to the device, and then waiting for a response packet. This, effectively, frees up the main CPU to do other things, rather than wrangle data. It also means that bottlenecks move.
So, it "requires" CPU support, just not in the way usually meant.
Disclaimer: I work on this with some very smart people @ IBM. Opinions are my own.
When a PCIe device is in CAPI mode, the PCIe protocol is used as a transport layer, but the CAPI protocol rides on top, and hardware in the CPU's PHB (the CAPP unit) and hardware in the accelerator (the PSL in this case) cooperate to present the common address space to the process and to the accelerator itself. [1] If a CAPI-capable card's plugged in to a non-CAPI-capable slot, it remains a PCI card. If a non-CAPI card's plugged in to a CAPI-capable system, it remains a PCI card. If both sides match on protocol versions and the kernel contains the cxl driver, the kernel will switch the slot into CAPI mode, and the CAPP unit and PSL effectively take over the PCI link on either side.
[1] http://events.linuxfoundation.org/sites/events/files/slides/... - see page 13+ for some GPU / NVLink materials, and page 24+ for CAPI materials (oh - and I worked on the product who's data is quoted on page 29 [2]!
[2] https://www.ibm.com/developerworks/community/blogs/fe313521-...
CAPI is much more flexible and interesting. It allows a CAPI-capable connected device access to a process's virtual memory. Essentially, you can extend the CPU's capabilities with CAPI. Usually this would be an FPGA (and the utility of FPGAs for machine learning is very much a research topic), but I could easily see a DSP being useful for voice recognition. GPUs can take advantage of it too, but ML work is usually just offloaded entirely to the GPU.
CAPI is very very cool and designed by some very smart people. I'm excited what people will do with CAPI and FPGAs.
http://www-304.ibm.com/webapp/set2/sas/f/capi/CAPI_POWER8.pd... is a good intro.
[Disclosure: I work on CAPI at IBM]
1. CAPI doesn't seem to get mentioned to much around here, but imagine an FPGA directly accessing some shared system memory. It's neat.