News for hackers. Stuff that matters.

A comment on 3Blue1Brown: Visualizing Attention, a Transformer's Heart. Parent comment

Re:3Blue1Brown: Visualizing Attention, a Transformer's Heart by faramarz on Sunday April 14, 2024 @11:54PM (#40036926)
it's not about a single point encapsulating a novel, but how sequences of such embeddings can represent complex ideas when processed by the model's layers.

each prediction is based on a weighted context of all previous tokens, not just the immediately preceding one.

[ Parent | on HN ]
Re:3Blue1Brown: Visualizing Attention, a Transformer's Heart by rollinDyno on Monday April 15, 2024 @01:53AM (#40037434)
That weighted context is the 12228 dimensional vector, no?

I suppose that when you each element in the vector weighs 16 bits then the space is immense and capable to have a novel in a point.

[ Parent | on HN ]
Re:3Blue1Brown: Visualizing Attention, a Transformer's Heart by causal on Monday April 15, 2024 @11:26PM (#40048088)
But if I understand correctly, GPT-4 reduces that to a 1536-dimensional vector. Roughly 1/8th. It's counterintuitive to me.
[ Parent | on HN ]
Re:3Blue1Brown: Visualizing Attention, a Transformer's Heart by faramarz on Wednesday April 17, 2024 @04:55PM (#40069972)
GPT-4 is configurable up to 96 layers, each running their own embeddings. I think it was a business choice to afford the compute while they scale.
[ Parent | on HN ]