Re:ChatGPT, Rot13, and Daniel Kahneman
by nverno on Friday December 09, 2022 @05:25AM
(#33919762)
Did you try entering one of those scrambled sentences into the tokenizer? It's not tokenizing words exactly- the doc says roughly ~100 tokens => 75 words.
[ Parent
| on HN ]
I just tried out of curiosity. The jumbled words get broken up into different tokens, the unjumbled words do not get split.