News for hackers. Stuff that matters.
Re:Ernie, China's ChatGPT, Cracks Under Pressure by jedberg on Thursday September 07, 2023 @01:44PM (#37422720)
When you learn Chinese, they teach you that the more complex words are just compounds of smaller words put into a single character. I imagine tokenizing it isn't much different than other languages -- for the complex characters you can tokenize it into its smaller parts.

There are no comments on this comment yet.