Re:OpenAI Tokenizer
by teruakohatu on Wednesday April 05, 2023 @05:53PM
(#35460896)
That is very helpful, thank you. I had not realised the latest models were now tokenizing number as 3 digit groups. Can you give any insight into why 3 digits?
[ Parent
| on HN ]
There are no comments on this comment yet.