Re:'I Worked on Google's AI. My Fears Are Coming True'
by skybrian on Tuesday February 28, 2023 @08:03PM
(#34977362)
There's a stateless function at the heart of it, but modeling it as a Markov chain isn't useful and I wish people would stop saying that.
A Markov chain models a system as a state machine where each state has outgoing transitions with probabilities that are independent of all the other states. When the state being modeled is up to 4k tokens of text, considering each possible state independently results in vanishingly small probabilities. You can't train a system that way.
A Markov chain models a system as a state machine where each state has outgoing transitions with probabilities that are independent of all the other states. When the state being modeled is up to 4k tokens of text, considering each possible state independently results in vanishingly small probabilities. You can't train a system that way.