Also, are they only comparing against other spiking neural nets instead of deep nets? If so, that doesn't tell much about whether these chips are useful for modern AI. I'd have to see their paper.
If you want to compare energy efficiency, you look at the hardware these models are running on (CPU, GPU, FPGA, ASIC), and see how much energy is spent by each of these chips to reach a certain accuracy (preferably on ImageNet).
You can also compare the speed of processing (significantly affected by image size): 1. How fast can you get to a certain accuracy? 2. How many images per second can you classify with that accuracy?
Strictly speaking, true. But one discovers early enough that you can screw up a number of things when training a model on something as simple as MNIST and the model will still work, because there's so much redundancy, the correct layers will just train their way around the ones that don't really work.
VGG trained for general purpose image classification with a decent sized input is a much better indicator of whether or not some ML hardware improvement is useful.
>"Further, it is up to 1,000 times more energy-efficient than general purpose computing required for typical training systems."
Wow. That is tremendous.