Re:Let's Think Dot by Dot: Hidden Computation in Transformer...
by wongarsu on Saturday April 27, 2024 @06:57PM
(#40184357)
And this works even if we assume the dots don't actually transfer information but just slightly mutate the state. First it tries a random approach, if that approach doesn't lead to a powerful result it emits a dot to try another random approach in the next round, until you get a sufficiently good path forwards.
Essentially a brute-force search. Which is a bit wasteful, but better than just blindly taking the first idea
Essentially a brute-force search. Which is a bit wasteful, but better than just blindly taking the first idea