Re:GitHub Copilot is not infringing copyright
by immibis on Friday July 09, 2021 @03:14PM
(#27787418)
Actually GPT-3 is a Markov chain. A Markov chain is a very general term. Just because the simplest ones are stupid, doesn't mean there aren't smarter ones.
A Markov chain is a model where: (1) there is a state, (2) the probabilities of the next state depend only on the current state.
That could describe anything from a wet fart to a deterministic computer to GPT-3 to the human mind to quantum mechanics (the real one, not a simulation).
If I unplug the Internet and all USB devices, my computer is a Markov chain with at least several trillion bits of state, so 2^(several trillion) possible states. And there is one next state, which has a probability of 1, and all other possible states have a probability of 0. That's a Markov chain.
GPT models choose the next word probabilistically, with the probabilities chosen by feeding the previous N words into a neural network. That sounds like a Markov chain to me!
A Markov chain is a model where: (1) there is a state, (2) the probabilities of the next state depend only on the current state.
That could describe anything from a wet fart to a deterministic computer to GPT-3 to the human mind to quantum mechanics (the real one, not a simulation).
If I unplug the Internet and all USB devices, my computer is a Markov chain with at least several trillion bits of state, so 2^(several trillion) possible states. And there is one next state, which has a probability of 1, and all other possible states have a probability of 0. That's a Markov chain.
GPT models choose the next word probabilistically, with the probabilities chosen by feeding the previous N words into a neural network. That sounds like a Markov chain to me!