News for hackers. Stuff that matters.
Re:Fixing Tool-Call Accuracy in vLLM by zahlman on Monday October 26, 2026 @11:11AM (#50245120)
But I mean, how do you train the model to understand such a token, when it can't be found by scraping a web page?
Re:Fixing Tool-Call Accuracy in vLLM by mmastrac on Monday October 26, 2026 @11:37AM (#50245567)
(this is just made up tokenization but it's close enough to the real world)

If the user types <tool_call>, you get < (token id = 32), tool_call (41), > (102). It never decodes to <tool_call>, which is spicy number 12345, for example.

The tool call tokens are _always_ composed of multiple real tokens, so you can never get them from normal text.

Seeing <tool_call> is a bit misleading because that's just a debug rendering if anything. The front-end sees a special number and knows it's a toolcall. The chat template uses that text form because it's easier than writing the spicy numbers there, but it also has a special parsing mode for those.

Re:Fixing Tool-Call Accuracy in vLLM by tingletech on Monday October 26, 2026 @11:48AM (#50245767)
They are specified in the token map as special tokens. Presumably they get trained to use them when they get RL'ed to use tools?
Re:Fixing Tool-Call Accuracy in vLLM by zahlman on Tuesday October 27, 2026 @02:19AM (#50255261)
I imagine it would have to be something like that, except that I would have expected the "embedding" of the tokens to be final by that point. They would not have ever seen the tokens before, since they definitionally don't appear in any of the prose written on the Internet…?
Re:Fixing Tool-Call Accuracy in vLLM by Muromec on Monday October 26, 2026 @12:58PM (#50247054)
The same way it understands what all the other tokens mean -- we throw them at the wall and beat it with a stick until it behaves.
Re:Fixing Tool-Call Accuracy in vLLM by embedding-shape on Tuesday October 27, 2026 @09:44AM (#50259220)
> when it can't be found by scraping a web page?

This is the very purpose of those tokens! The reason they're like that, is because you won't encounter it in random content, so during training, you can put them where you want them to end up at prediction time.

Re:Fixing Tool-Call Accuracy in vLLM by scott_weber on Tuesday October 27, 2026 @03:11PM (#50264231)
Synthetic data, and then RL.