If we are being pedantic, transformers operate on integer vectors, and in the process turn them into high dimensional matrices.

I actually had to check the publish date on this. The first few paragraphs are absurd.

But they are not transparently symbolic.

They are not but there’s a lot of lifting being done by assuming a “hypervector” (an embedding, to literally everyone else) for a concept. Does such a separable embedding exist and do you have a good way to find it? Seems like one of those things where, oops, it’s just worse than gradient descent on massive functions.

I had the same reaction. The whole time I was expecting a paragraph to explain why this is fundamentally different from binary classification techniques like SVM, which explicitly uses a hyperplane in high-dimensional space to divide the classes.

Hah, same here. How is [colour, shape] or [blue, red, square, circle] not a vector? How does a NN work if not by operating on vectors/matrices/tensors?

This Quanta Magazine is absolute horseshit. Might as well just remove all that text and link to the author's paper. But hey, it makes for a nice article with a progress bar that animates as you scroll up and down.

The papers:

https://neuroscience.berkeley.edu/wp-content/uploads/2016/05...

https://redwood.berkeley.edu/wp-content/uploads/2022/05/kane...

If they linked the papers, it would be too obvious how poorly they described what they were about.