I didn't say it's not valid, or that it shouldn't have been done. I'm just saying "NN still doesn't help them beat iPhone" is not a relevant comment regarding this test. To go with car analogy it's close to an article about an slightly improved design of one part of fuel injection in new ABC and someone commenting "look, an XYZ from last year still can go faster without it".

The test is a valid observation, but is irrelevant to this article, or to the technology.

Why measure something that doesn't show up in a standard user's test? Here are some examples: lower power usage, lower latency of small operations (rather than throughput of large actions), smaller design (saving chip space), etc. And finally - if this is a good technology, Apple can use it too and get even faster.

That is a great analogy. That test in the video didn't really mean anything towards the overall speed of the device. It's interesting to see, but I how this test would change over n trials.

You should do that, but not only that. If you just test the way the product is being used, you have a million possible reasons why it could be slow. If you just want to know whether slapping a neural network on the chip is a good idea or not, it's best to test that in such a way as to exclude as many confounding factors as possible.

In general: because how a piece of hardware (or set of hardware) is used today isn't necessarily how it's going to be used tomorrow. Faster hardware allows for use-cases that weren't possible prior.

However, when we're talking about phones or consumer devices, how it's used today is the only thing that's relevant. It will be obsolete in just a couple of years.

I think the parent point is still valid. Look at Pokemon Go. Average user wouldn't care about AR performance, all that high gps accuracy or gfx performance in unity. Suddenly, there are millions of players who are choosing their hardware based on that. If you are planning for today's usage you you may be lagging.