Re:GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Text
by sp332 on Sunday December 03, 2023 @03:13PM
(#38510335)
If you put it into the tokenizer https://platform.openai.com/tokenizer you can see that it helps in some places but not in others. It pulled out "SEA/OF/TRAN/QU/ILITY", but I think it broke up every instance of the word "THE".
[ Parent
| on HN ]
There are no comments on this comment yet.