I was personally quite appalled that so many people believed this was what it was doing, but I suppose since the source was a DeepMind researcher, it lends a lot of credibility.

By the same reasoning, I hope people do realise that there are computations that are just impossible given the finite depth in a Transformer. The only possible way to overcome this in the current paradigm is the Chain-of-Thought related methods.