News for hackers. Stuff that matters.
Re:Fixing Tool-Call Accuracy in vLLM by jesdo on Monday October 26, 2026 @10:10AM (#50244024)
To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.

I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.

Re:Fixing Tool-Call Accuracy in vLLM by zahlman on Monday October 26, 2026 @10:40AM (#50244541)
So to be clear, you can't prompt-inject these?

... How did they get into the training data?

Re:Fixing Tool-Call Accuracy in vLLM by mmastrac on Monday October 26, 2026 @10:56AM (#50244854)
They are a special token that you can't just type out in text. It _renders_ as text, but cannot be input into the system as text. The model understands the difference between the textual toolcall token and the "special" one that's really just a spicy number.
Re:Fixing Tool-Call Accuracy in vLLM by zahlman on Monday October 26, 2026 @11:11AM (#50245120)
But I mean, how do you train the model to understand such a token, when it can't be found by scraping a web page?
Re:Fixing Tool-Call Accuracy in vLLM by mmastrac on Monday October 26, 2026 @11:37AM (#50245567)
(this is just made up tokenization but it's close enough to the real world)

If the user types <tool_call>, you get < (token id = 32), tool_call (41), > (102). It never decodes to <tool_call>, which is spicy number 12345, for example.

The tool call tokens are _always_ composed of multiple real tokens, so you can never get them from normal text.

Seeing <tool_call> is a bit misleading because that's just a debug rendering if anything. The front-end sees a special number and knows it's a toolcall. The chat template uses that text form because it's easier than writing the spicy numbers there, but it also has a special parsing mode for those.

Re:Fixing Tool-Call Accuracy in vLLM by tingletech on Monday October 26, 2026 @11:48AM (#50245767)
They are specified in the token map as special tokens. Presumably they get trained to use them when they get RL'ed to use tools?
Re:Fixing Tool-Call Accuracy in vLLM by zahlman on Tuesday October 27, 2026 @02:19AM (#50255261)
I imagine it would have to be something like that, except that I would have expected the "embedding" of the tokens to be final by that point. They would not have ever seen the tokens before, since they definitionally don't appear in any of the prose written on the Internet…?
Re:Fixing Tool-Call Accuracy in vLLM by Muromec on Monday October 26, 2026 @12:58PM (#50247054)
The same way it understands what all the other tokens mean -- we throw them at the wall and beat it with a stick until it behaves.
Re:Fixing Tool-Call Accuracy in vLLM by embedding-shape on Tuesday October 27, 2026 @09:44AM (#50259220)
> when it can't be found by scraping a web page?

This is the very purpose of those tokens! The reason they're like that, is because you won't encounter it in random content, so during training, you can put them where you want them to end up at prediction time.

Re:Fixing Tool-Call Accuracy in vLLM by scott_weber on Tuesday October 27, 2026 @03:11PM (#50264231)
Synthetic data, and then RL.
Re:Fixing Tool-Call Accuracy in vLLM by pyrolistical on Monday October 26, 2026 @11:23AM (#50245329)
Why can’t user request be spicy numbers so prompt injection is impossible
Re:Fixing Tool-Call Accuracy in vLLM by Muromec on Monday October 26, 2026 @12:57PM (#50247023)
It can happen that you would be able to inject them if there is a bug, same way sql injections work.

You have all the same pieces -- untrusted input on the left and the thing that accepts a stream of tokens on the right. In the middle could be anything, especially when vibecoded.