Re:Let's Think Dot by Dot: Hidden Computation in Transformer...
by dist-epoch on Saturday April 27, 2024 @06:12PM
(#40184064)
You can transfer some state just through dots. The dot count could mean "the first n ideas do not work, analyze the n+1 one, if that's bad, emit another dot"
[ Parent
| on HN ]
Re:Let's Think Dot by Dot: Hidden Computation in Transformer...
by wongarsu on Saturday April 27, 2024 @06:57PM
(#40184357)
And this works even if we assume the dots don't actually transfer information but just slightly mutate the state. First it tries a random approach, if that approach doesn't lead to a powerful result it emits a dot to try another random approach in the next round, until you get a sufficiently good path forwards.
Essentially a brute-force search. Which is a bit wasteful, but better than just blindly taking the first idea
Re:Let's Think Dot by Dot: Hidden Computation in Transformer...
by im3w1l on Saturday April 27, 2024 @07:29PM
(#40184571)
Kind of like trying nonces until you find one that gives you a hash with lots of leading zeros? Dotchain?
Re:Let's Think Dot by Dot: Hidden Computation in Transformer...
by kevindamm on Sunday April 28, 2024 @12:40AM
(#40186062)
Chain of Dot.
Re:Let's Think Dot by Dot: Hidden Computation in Transformer...
by diziet on Sunday April 28, 2024 @06:18AM
(#40187453)
I believe the model is trained to always output the same number of dots. Additionally, the way LLMs generate output tokens are not really in in line with transferring state this way.