Batch 1 inference on convnets is key for us internally but training does work pretty well. The underlying machinery can do much more. Here's a blog post that talks about how it works with some links to more detailed docs & the actual implementations:

http://vertex.ai/blog/tile-a-new-language-for-machine-learni...

Two of the big motivators for opening the code were 1) giving students taking the popular courses a way to get started with GPU in whatever machine they've got (recent Intel GPUs in say a MacBook Air are enough) and 2) giving researchers a platform where it's simple to add efficient GPU-accelerated ops.

For scale on #2 check out the entire implementation of convolution:

https://github.com/plaidml/plaidml/blob/master/plaidml/keras...