This is what was intensively discussed by first trial users, and later admitted by google tpu team themselves on hot chips 29

Even with monsterous HBM2 memory, they still have it.

It is probably hard to predict what matrix set to prefetch when you deal with a neural net. So you have cache misses there too