News for hackers. Stuff that matters.
Re:Teaching ChatGPT to Speak My Son’s Invented Language by shagie on Tuesday April 11, 2023 @12:43PM (#35527657)
While its more token costly, the tokenization of numbers gets weird.

Taking "123" it becomes [123]. "1234" becomes [12][34]. "12345" becomes [123][45].

Consider using spaces between each digit to force them to be distinct tokens.

"1 2 3 4 5" is tokenized as [1][ 2][ 3][ 4][ 5]

It may then be possible to work with the digits more easily because they're not getting tokenized in potentially odd ways that are harder to manipulate.

There are no comments on this comment yet.