News for hackers. Stuff that matters.
Re:ChatGPT, Rot13, and Daniel Kahneman by nverno on Friday December 09, 2022 @05:25AM (#33919762)
Did you try entering one of those scrambled sentences into the tokenizer? It's not tokenizing words exactly- the doc says roughly ~100 tokens => 75 words.
Re:ChatGPT, Rot13, and Daniel Kahneman by pa7x1 on Friday December 09, 2022 @09:33AM (#33921572)
I just tried out of curiosity. The jumbled words get broken up into different tokens, the unjumbled words do not get split.

https://imgur.com/a/a3zmkIv