Despite this impressive achievement, the author does not appear to have the most solid theoretical background.

Some explanations (e.g. regarding UTF encoding) are not entirely correct and use uncommon terminology (e.g. alphabet instead of vocabulary).

Also, about the non-determinism issue, there was a post some time ago and that comes from the way the GPU does the calculations, something something floating point something.

So of course the algorithm is deterministic, but the real-life implementation isn't.

Floating point addition, for example, is not associative, so the order of taking a sum affects the result. If the summation were sequential and single threaded, it would be deterministic. But it happens in parallel, so timing variations affect the result.

But there is probabilistic sampling that happens (see "temperature").

> Floating point addition, for example, is not associative, so the order of taking a sum affects the result. If the summation were sequential and single threaded, it would be deterministic. But it happens in parallel, so timing variations affect the result.

In this sense, I don't think it's fair to say floating point math is non-deterministic, as much as parallel computation is non-deterministic. FP behaves in unexpected ways, but the same order of operations always yields the same unexpected results (except on Pentium 1).

Electricity, Cars, and Gas were once upon a time a luxury as well - reserved to those who could afford them or had unique access / credentials / skills. The people who were able to simplify and spread the advanced tech to the common person became Billionaires.

when talking about a set of characters, alphabet is a commonly used term https://cs.stackexchange.com/search?q=string++alphabet+size

Sure, but if your doing work in machine learning that’s generally not the terminology used, hinting that this isn’t the area the author specializes in (which isn’t a bad thing, but take their explanations with a grain of salt).

> and almost in a childish way.

No. You've got to have a solid background in computer science to even start to understand fully this article.

Even the title itself is not accessible to 99% of humans.

Count me in as one of the 99%

Parent obviously means childish as in "simple and easy manner" not as "something 10 year olds will understand".

This sorcery didn't start a year ago. The model being described in the article is GPT-2 which was released in early 2019.