Re:GPT-3.5 crashes when it thinks about useRalativeImagePath...
by npsomaratna on Sunday January 21, 2024 @11:51PM
(#39086352)
Correct me if I'm wrong—but we don't know if GPT-4 uses the same tokenizer as GPT-3.5, right?
Re:GPT-3.5 crashes when it thinks about useRalativeImagePath...
by minimaxir on Sunday January 21, 2024 @11:55PM
(#39086372)
OpenAI's web tokenizer demo confirms it: https://platform.openai.com/tokenizer
Re:GPT-3.5 crashes when it thinks about useRalativeImagePath...
by shawntan on Monday January 22, 2024 @02:46AM
(#39087305)
Exactly this. The tokens generated should always be valid, unless some post-processing layer between the model's output and the user interface detects for some keywords which it would prefer to filter out. In which case I suppose there is another commonly seen error message that appears?
Re:GPT-3.5 crashes when it thinks about useRalativeImagePath...
by SirSegWit on Monday January 22, 2024 @03:00AM
(#39087364)
Not really, right? There are a ton of special tokens, like start of sequence etc., so what happens if there are two start of sequences predicted? It's a valid token but cannot really be turned into something sensible, so it throws an error when converting tokens to plain text?
Re:GPT-3.5 crashes when it thinks about useRalativeImagePath...
by minimaxir on Monday January 22, 2024 @03:02AM
(#39087377)
Special tokens are handled by the application, not the model. They are still output before then.
Re:GPT-3.5 crashes when it thinks about useRalativeImagePath...
by londons_explore on Monday January 22, 2024 @03:57AM
(#39087634)
I suspect more likely this token is simply blacklisted after the r/counting incident - ie. any response containing it will now return an error.
Re:GPT-3.5 crashes when it thinks about useRalativeImagePath...
by wycy on Monday January 22, 2024 @08:16AM
(#39089163)
What was the r/counting incident?
Re:GPT-3.5 crashes when it thinks about useRalativeImagePath...
by sgerenser on Monday January 22, 2024 @09:05AM
(#39089644)
Presumably this: https://www.vice.com/en/article/epzyva/ai-chatgpt-tokens-wor...
That isn't how LLMs generate tokens. Each step outputs a logit for each possible token in the tokenizer (100k in the case of GPT-3.5), then softmaxes the logits to covert them into probabilities, and samples from them depending on temperature to get the token to be used.
It's possible something in the tokenizer BPE merge process breaks due to the rare token, which can be verified offline using tiktoken. But if GPT-4 works, and since GPT-3.5 and GPT-4 use the same tokenizer, then that's likely not the issue.