News for hackers. Stuff that matters.
Re:OpenAI Tokenizer by dschnurr on Wednesday April 05, 2023 @05:32PM (#35460648)
Hi folks – I work at OpenAI and helped build this page, awesome to see it on here! Heads up that it's a bit out of date as GPT4 has a different tokenizer than GPT3. I'd recommend checking out tiktoken (https://github.com/openai/tiktoken) or this other excellent app that a community member made (https://tiktokenizer.vercel.app)
Re:OpenAI Tokenizer by teruakohatu on Wednesday April 05, 2023 @05:53PM (#35460896)
That is very helpful, thank you. I had not realised the latest models were now tokenizing number as 3 digit groups. Can you give any insight into why 3 digits?
Re:OpenAI Tokenizer by resters on Wednesday April 05, 2023 @08:25PM (#35462462)
Was the purpose of the page and post to generate comments that can be used as training data?
Re:OpenAI Tokenizer by lemming on Wednesday April 05, 2023 @09:38PM (#35462995)
Are there plans to release tokenisers for other platforms? I'm accessing the OpenAI API from Clojure, and it would be really nice to have a JVM version so I can estimate token use before sending.
Re:OpenAI Tokenizer by lowefk on Wednesday April 05, 2023 @10:56PM (#35463558)
I wasn't aware that GPT-3 and GPT-4 use different tokenizers. I've read https://github.com/openai/openai-cookbook/blob/main/examples... and misinterpreted "ChatGPT models like gpt-3.5-turbo and gpt-4 use tokens in the same way as older completions models, ..." as GPT-3 and GPT-4 using the same tokenizer except for im_ tokens. Now I can see so many improvements, including the encoding of whitespaces and digits.
Re:OpenAI Tokenizer by egorfine on Thursday April 06, 2023 @10:49AM (#35469053)
Hey it seems that UTF-8 support is broken on the page.

Test phrase could be something like "Жизнь прекрасна и удивительна" ("Life is great" in russian).

I make an assumption that this is the implementation on the page that is broken, not the actual tokenizer. The reason: russian works perfectly in GPT-3 which I guess wouldn't be the case with a tokenization as presented on the page.

Re:OpenAI Tokenizer by dqbd on Sunday April 09, 2023 @05:21PM (#35506594)
Author here, you are correct! The issue here is due to the fact that a single user-perceived character might span into multiple tokens. This should be fixed now.
Re:OpenAI Tokenizer by egorfine on Monday April 10, 2023 @05:00AM (#35510440)
Hey. Thank you! However has the fix not been deployed yet? Still shows broken UTF-8.

> a single user-perceived character might span into multiple tokens

Is this the way it works as designed or is this a bug?