News for hackers. Stuff that matters.
Re:3Blue1Brown: Visualizing Attention, a Transformer's Heart by mjburgess on Monday April 15, 2024 @08:33AM (#40039739)
How do you think the moves are represented?

All abstract games of this sort are just sequences of bit patterns, each pattern related to the full legal space by a conditional probability structure (or, equivalently, as set ratios).

Strip away all the NN b/s and anthropomorphic language and just represent it to yourself using bit sets.

Then ask: how hard is it to approximate the space from which these bit sets are drawn using arbitrarily deep conditional probability structures?

it's trivial

the problem the author sets up about causal structures in the world cannot be represented as a finite sample of bit set sequences -- and even if it could, that isnt the data being used

the author hasn't understood the basics of what the 'world model' problem even is

There are no comments on this comment yet.