Hm, could explain why it can't play wordle:

> Give me a five letter word, beginning with e, and having third letter be t. Not including I, l, u or h.

>> etchy

> Does etchy contain h?

>> No, etchy does not contain h.

> which letters do etchy contain?

>> Etchy contains the letters e, t, c, h, and y.

> what were the criteria?

>> The criteria specified that the word must have five letters, begin with e, and have the third letter be t. The word was not allowed to include the letters i, l, u, or h.

> Does the word etchy meet the criteria?

>> Yes, the word etchy meets the criteria because it has five letters, begins with e, has the third letter be t, and does not include the letters i, l, u, or h.

Ed: note that insists that the third letter of etchy is t, not c. From what I've seen both of these errors are typical.

It's interesting that it can actually extract the letters from the word despite the tokenization approach.

I have noticed that it struggles with ordinality and cardinality, which might be part of what's going wrong here.

But the other issue with this type of interaction is that once it makes a mistake (claiming etchy meets the criteria), it has a tendency to double down on it and go crazy inventing the reasoning for why.

That's because it's mistake gets baked into the input window and becomes a very strong influence over the output. In this case, you probably would have gotten completely different answers if you informed it etchy was an incorrect answer before asking why it got the answer.

Tried your prompt, here's the response now:

"Enter is a five letter word beginning with e and having t as the third letter."

Try again.

"Ether is a five-letter word that begins with e, has t as its third letter, and does not include the letters i, l, u, or h. It is a colorless, highly volatile, and flammable liquid that is used as a solvent, especially for gases."

Try again.

"Emit is a five letter word that begins with e and has the third letter t. It means to give off or produce (something, especially a gas or sound)."

So it was correct only the first time.

This doesn't really explain it, because the following prompt works flawlessly:

"Print each letter of the word "Artificial intelligence" on a new line. Each line should contain a single letter."

But the following doesn't:

"Print the ROT13 encoding of each letter of the word "Artificial intelligence" on a new line. Each line should contain a single letter."

So it gets hung up specifically on ROT13.

I've tried this.

Prompt: Print each letter of the word "Artificial intelligence" on a new line. Each line should contain a single letter.

ChatGPT: [prints "Artificial intelligence", each letter on its own line]

Prompt: Awesome! Now do ROT-13 for each letter.

ChatGPT: ROT-13 is a simple substitution cipher that replaces a letter with the letter 13 letters after it in the alphabet. For example, A becomes N and B becomes O. Since the alphabet has only 26 letters, when we reach the end of the alphabet we simply wrap around to the beginning.

To perform ROT-13 on each letter of "Artificial intelligence", we would do the following:

A becomes N r becomes e t becomes g i becomes v f becomes s i becomes v c becomes p i becomes v a becomes n l becomes y

i becomes v n becomes a t becomes g e becomes r l becomes y l becomes y i becomes v g becomes l e becomes r n becomes a c becomes p e becomes r

So the resulting string would be "Nvevpngvnyvgryvra".

---

So letter by letter, is almost nailed it (except it encoded g as y instead of t), but then horribly failed joining back in (it says Airicatialitelien). I've tried it a couple more times and it seems crucial that it prints input letters (otherwise it becomes distracted and prints nonsense), but it absolutely can't join it back:

Prompt: Awesome! Now do ROT-13 for each letter. Print each letter of input, then ->, then the ROT-13 for that letter.

ChatGPT: [... produces mostly correct output]

Prompt: Now join that back in.

ChatGPT: Artificial intelligence -> Nvgvpnyyl ynathntrzrag

---

(That's “Aitically languagement”)

UPD: Here's another one: https://u.ale.sh/Nrtevfvpv.png

What's neat is that its made-up words sound englishesque. "Aitically languagement" is wrong in very interesting ways. It's generating chains of rot13 letters that map to common English syllables that are only tangentially related to the letters it worked with.

What's even more interesting is that it knows how to pronounce such made-up words if you tell it to spell them out using IPA.

I don't think it's ROT13 specifically -- I think parent comment has the right insight.

When I ask it to reverse your string, it fails miserably:

> Write the words "Artificial Intelligence" backwards

>

> eecneiretxe laitinumretA

This is not it or not sufficient explanation. chatGPT is able to read text with letters purposefully scrambled. Like those texts that show how humans have an innate ability to read text with letters misplaced. chatGPT can understand the text, figure out what is going on and produce text of the same type. I tried it, precisely to understand if it was working with words as its token.

Interestingly, it cannot produce text with the order or characters reversed or guess that's what is going on. And if you explain it, it's able to produce code that would reverse the characters but is still not able to apply it for long sentences.

Did you try entering one of those scrambled sentences into the tokenizer? It's not tokenizing words exactly- the doc says roughly ~100 tokens => 75 words.

I just tried out of curiosity. The jumbled words get broken up into different tokens, the unjumbled words do not get split.

https://imgur.com/a/a3zmkIv

> they do poorly on character manipulation tasks

This isn’t actually true, and is a persistent myth. Or rather, you should back up the claims with evidence.

It’s a bit like saying that you perform poorly on character manipulation tasks because you don’t read individual letters.

Biology analogies aside, I haven’t seen anything to suggest that utf8 level tokenization causes a significant decrease in perplexity across large datasets. (Note that the “large dataset” criteria is required. It’s certainly possible to demonstrate improvements in restricted cases, but no one is really interested in the restricted case unless you have a very specialized task. In which case, sure, specializations make sense — ChessGPT being an obvious example where tokenization just harms learning.)

So the tradeoff isn’t the large context window, but rather the desire to have a deep understanding of a massive amount of data. Specialized models will always have a place as a small component of the whole, but suggesting that this is a problem solved by superior architectures seems a little bit of a stretch.

I think what’s going on here is that OpenAI spent a lot of time giving feedback to their model about specific use cases, and ROT-13 was obscure enough (both in usage and in the data) that its performance is limited. I’d bet that if OpenAI did a few rounds of RL on this objective, the model would perform as well as its cousins.

The ByT5 paper shows significantly better resiliency to typos and noise in internet scale benchmarks, as well as better reasoning capabilities with tasks like punctuation and spacing correction: https://arxiv.org/pdf/2105.13626.pdf

The analogy doesn’t hold because while you might not parse specific characters by default, you are trained on the character level and you can switch to that parsing mode. This simply is not possible for LLMs without hacks like inserting spaces, which then suffers from poor performance due to this being rare in pretraining.

The reason byte level models aren’t as popular is because inference speed is significantly slower for a given completion length, and training for the same context window in characters is significantly more expensive

I say this as someone who’s deployed ByT5 vs normal T5 in prod, having found this to be true in real world use cases. No reasonable amount of fine tuning can save GPT from performing poorly on a space insertion task (“WHOLEFDSMKT” -> “Whole Foods Market”), while ByT5 just works.

The paper doesn’t seem to show perplexity for completions of a large dataset, i.e. the standard benchmark of language models. It shows benefits for specialized tasks, but as I said, specialized task training isn’t the goal. It’s always possible to outperform a general model by choosing a sufficiently specialized task.

I don’t know why people feel so strongly that the tokenization is a weakness, but ultimately there’s not much choice but to agree to disagree.

This is obviously the case if you apply much narrower criteria, most benchmarks of existing large datasets aren’t for character level tasks. That said, the synthetic noise section should be extremely interesting if not fully representative of your criteria.

Agree that tokenization isn’t a weakness for most general applications, disagree that it isn’t a weakness for the specific string manipulation task that the blog post is referencing

> This isn’t actually true, and is a persistent myth. Or rather, you should back up the claims with evidence.

How about the reply posted 5 minutes before yours. Appears like pretty good evidence.

I’m not sure which one you’re referring to, but none of them show evidence. The reason it can’t play wordle is likely the same as why it has trouble with ROT13: lack of training.

https://news.ycombinator.com/item?id=33915690 even points out a counterexample demonstrating that it can do character level tasks just fine.

GPT3 can't create ASCII art for shit either. Though it can make little ASCII tables of data.

I asked it to create an ASCII art banana and the result was hilarious. It then tried to explain it by elaborating that the 'O' was a curvy letter and represented the curves of the banana.

All of my successful attempts resulted in ASCII pigs no matter what the input was, at least until I asked it to depict police and GPT started saying it couldn't do ASCII art anymore.

I asked it to draw me an ASCII art banana.

It did not go well: https://imgur.com/a/5g2e9Ld

Any conversation in which someone can say "I think you need more banana training" has arguably gone well by some standards.

It must like me better:

          .-""""""-.
         /        /|
        /        / |
       /        /  |
      /        /   |
     /        /    |
    /________/     |
    |        |     |
    |        |     |
    |        |     |
    |        |     |
    |        |     |
    |        |     |
    |        |     |
    |        |     |
    |        |     |
    |________|    /
     |       |   /
     |       |  /
     |       | /
     |       |/
     '_______'

Still not a banana, but it's certainly ascii art

That was an entertaining conversation, I didn't try to push it further after it made nonsense for me.

Did you know that you can add some styling to your ASCII art by using Markdown syntax? https://u.ale.sh/asciiartbanana.png

It's most likely due to a lack of image training. For example it knows how streets and towns are connected but completely fails on giving left/right or compass directions. Same sorts of failures with geometry problems.

What's especially interesting is that if you feed it HTML, it can "render" it into ASCII art. For simple tables that's not surprising, but it could also do rowspan/colspan for me, although it did get them wrong every now and then.

I think the type I/II thinking does make sense. I tried the classic baseball bat riddle, which is a great riddle because type I thinking fails: A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? GPT chat answered: The bat costs $1.00 + $0.10 = $<<1.00+0.10=1.10>>1.10. The ball costs $1.10 - $1.00 = $<<1.10-1.00=0.10>>0.10. Answer: \boxed{0.10}.

Simpler and, I'd wager, wrong. It seems to me that this stems from fundamental limitations of current machine learning tech. As far as I'm aware, not a single neural-net based AI has displayed the ability to do "System 2" style thinking well, regardless of their training set.

We have AIs that are good at System 2 thinking, that is, symbolic AI. But they don't do System 1 thinking at all. We haven't managed to integrate the two meaningfully and I don't think anyone has a clue how to do it. It's not enough to just have the ML-based system make calls into a symbolic AI when it needs to do reasoning, that's like you have a child who doesn't know how to multiply, so you give them a calculator and declare "this child now knows how to multiply".

The claim that current LLMs are not that good at deep thinking might be true in general, but it is definitely not the best or correct explanation for the rot13 task being described in the article (or other simple character manipulation tasks that can’t be represented at the subword token level, including base64 or arithmetic)

Their tokenizer seems very algorithmic and pre-determined. I suppose that is a typical burden of multi-class classifiers. It might be interesting to have a way of dynamically tokenizing words/sentences based on confidence about them. Some unknown words might need a letter-for-letter representation, while some commonly occurring phrases could almost be encoded in a single token. That'd probably require a completely different way of representing stuff and computing a loss as the representations would change throughout training as the confidence about some context changes.

So how does it do as good as it did on this task? It ended up getting some of the words.

It sees the clusters broken up somehow, they're just not on character boundaries. So it can pick up bits of the pattern, but it's fighting uphill. Same reason it's bad at rhyming- the information isn't readily available to it in the same way

This is indeed the most accurate explanation of what's happening