Not 100M I guess, but this is fresh from today's arXiv:

"Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens"

https://arxiv.org/pdf/2401.17377.pdf

For the first few years of smartphones markov chains were used for autocorrect.

Google did it. Amazon did it. Plenty of others did. What do you think they were doing before recurrent neural networks?

100 million USD?

No.

I'd believe $40 for the energy cost, $120 for very slightly increased wear on their hard drives, and $400 for one engineer's 20% time project for one week.

And that's if it was trained on Google's entire internet cache rather than, say, just a Wikipedia snapshot from 2004, which sounds like the kind of thing that Google might have set as a pre-interview coding challenge.

The trick with Markov chains is that you don't need to.

Markov Chains are dead simple. There's not really a "training" as much as it's simply reading data and collecting statistics.

They're so simple that you can probably build one nearly as fast as you can read the training data.

I think it would take some spectacularly bad engineering to be that wasteful. It would need to be so inefficient that getting ChatGPT to write the code won't be bad enough.

It would make no sense; they are not powerful models worthy of the spend.