News for hackers. Stuff that matters.

dqbd

A tuple does have a upper limit of 10 000 elements, which means that with this approach we can count to 10 000 at most.

Another approach which I tried is to do arithmetics on digits directly, storing digits in a tuple instead, but the code is not as elegant as the tuple one

https://github.com/dqbd/ts-math-evaluate/blob/main/src/math/...

Re:OpenAI Tokenizer on Sunday April 09, 2023 @05:21PM (#35506594)
Author here, you are correct! The issue here is due to the fact that a single user-perceived character might span into multiple tokens. This should be fixed now.