News for hackers. Stuff that matters.
Re:Fixing Tool-Call Accuracy in vLLM by jesdo on Monday October 26, 2026 @09:29AM (#50243338)
I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
Re:Fixing Tool-Call Accuracy in vLLM by hypfer on Monday October 26, 2026 @10:58AM (#50244899)
So if I understand it correctly, the tool calls are not actually done by the model?

In the sense that the model manifests them, but they're not actually a result of the model doing anything but the inference engine filtering down the anything only to valid moves?

Could something like that also be used in other contexts?

I heard about how people dynamically steer models by massaging the weights a bit, but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?

Re:Fixing Tool-Call Accuracy in vLLM by NitpickLawyer on Monday October 26, 2026 @11:53AM (#50245873)
> but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?

Yes, this is what constrained decoding means. You constrain the model to only output from a set of "allowed" tokens (a.k.a. grammars).

So, if you use tool use with constrained decoding, you can be certain that the model will always choose from a list of approved tools (i.e. if tools=['read','write'], it cannot output 'execute', even if in other sessions it could, if execute was in the approved tools).

The tricky problem is that certain models work / have been trained in certain ways. And constraining them can lead to "invisible" problems. Constrained decoding only guarantees perfect syntax, not perfect semantics (i.e. it could write nonsense in the correct file).

> Could something like that also be used in other contexts?

Sure. You can build pretty cool grammars and constrain the model to do cool stuff, akin to "tool use" but more free hand. You could, for example, make it always emit a 10 point plan. For any prompt, it will give you a fixed syntax 10 point plan, always. And the plan will be constant with a plan_title and 10{plan_item, item_text}. But, as above, there's no guarantee that the individual items in the plan would make any sense... (i.e. step7 in a pasta making recipe would be "add glue").

Re:Fixing Tool-Call Accuracy in vLLM by hypfer on Monday October 26, 2026 @11:56AM (#50245931)
The point I was getting at was more of a question of "Okay, how can this be used adversarially?" or "How can we break this?".

"What can a malicious inference engine do"

Something like that

Re:Fixing Tool-Call Accuracy in vLLM by NitpickLawyer on Monday October 26, 2026 @12:03PM (#50246063)
Sorry, I'm not following. Constrained generation is something that helps you, the caller, to get back a message that looks a particular way. I'm having trouble seeing how this would work in an adversarial way. Perhaps you can elaborate?

A malicious inference engine can already return anything it wants, it doesn't need constrained decoding on the LLM side. If your client blindly runs commands in an unprotected environment, you're already pwned. They don't even need to run an LLM, they could just randomly add "tool_response:{'execute':'rm -fR *'} and that'd be it...

Re:Fixing Tool-Call Accuracy in vLLM by hypfer on Monday October 26, 2026 @12:07PM (#50246153)
That was exactly my question.

So the fact that got me curious is that one can steer the output, which, at least to my understanding of the tech, breaks the assumption that one can not do that.

So the thought is "assumptions are being broken here. Which ones? And what are the implications of that?"

Re:Fixing Tool-Call Accuracy in vLLM by Muromec on Monday October 26, 2026 @12:48PM (#50246879)
You should treat inference provider as a C2 node of a rootkit. As long as it's your c2 or it's aligned with you somewhat, you are fine
Re:Fixing Tool-Call Accuracy in vLLM by FearNotDaniel on Monday October 26, 2026 @11:28AM (#50245422)
Correct, here's an implementation example using the Claude SDK [0] but it's the same approach on other platforms: you tell the model what tools it has available, it sends back a response saying it wants to use a specific tool with given parameters, your code executes that tool then sends the output back to the model on the next turn.

[0] https://platform.claude.com/docs/en/agents-and-tools/tool-us...

Re:Fixing Tool-Call Accuracy in vLLM by hypfer on Monday October 26, 2026 @11:49AM (#50245795)
I think I might've phrased something in ways that were confusing to people, because now two people have explained tool calls to me for some reason?
Re:Fixing Tool-Call Accuracy in vLLM by FearNotDaniel on Monday October 26, 2026 @02:53PM (#50249055)
Just trying to be helpful. I couldn’t see any other way to interpret the question below except as someone who doesn’t know how tool calls get done and who might therefore value a little insight. Didn’t mean to be patronising, apologies if I misinterpreted your question.

> So if I understand it correctly, the tool calls are not actually done by the model?

Re:Fixing Tool-Call Accuracy in vLLM by jesdo on Monday October 26, 2026 @11:15AM (#50245204)
Exactly. The LLM is your next-token predictor, which output text and tool-calls. These calls get picked up by your client/harness (e.g., opencode) which executes them and sends output back to the LLM.
Re:Fixing Tool-Call Accuracy in vLLM by hypfer on Monday October 26, 2026 @11:49AM (#50245799)
Yeah I know what a tool call is.
Re:Fixing Tool-Call Accuracy in vLLM by febed on Monday October 26, 2026 @11:18AM (#50245244)
Would this technique work with inference optimization techniques like DFlash and MTP? In other words does it slow down things to the point where its just better to retry?
Re:Fixing Tool-Call Accuracy in vLLM by jesdo on Monday October 26, 2026 @11:37AM (#50245573)
Yes, constrained decoding and speculative decoding work together, although they need to be carefully integrated and incorrect integrations give result in bugs (see point 3 under Section "Differences in Constrained-Decoding Behavior" in the blog post). Performance overhead does not generally seem to be a large problem. Of the speculative tokens, those that are not allowed by the grammar get discarded.
Re:Fixing Tool-Call Accuracy in vLLM by ivanjermakov on Monday October 26, 2026 @09:58AM (#50243825)
Do you know where <||> syntax is coming from? I find it very odd that existing markup wasn't used and there is very little info online about it.
Re:Fixing Tool-Call Accuracy in vLLM by jesdo on Monday October 26, 2026 @10:10AM (#50244024)
To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.

I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.

Re:Fixing Tool-Call Accuracy in vLLM by zahlman on Monday October 26, 2026 @10:40AM (#50244541)
So to be clear, you can't prompt-inject these?

... How did they get into the training data?

Re:Fixing Tool-Call Accuracy in vLLM by mmastrac on Monday October 26, 2026 @10:56AM (#50244854)
They are a special token that you can't just type out in text. It _renders_ as text, but cannot be input into the system as text. The model understands the difference between the textual toolcall token and the "special" one that's really just a spicy number.
Re:Fixing Tool-Call Accuracy in vLLM by zahlman on Monday October 26, 2026 @11:11AM (#50245120)
But I mean, how do you train the model to understand such a token, when it can't be found by scraping a web page?
Re:Fixing Tool-Call Accuracy in vLLM by mmastrac on Monday October 26, 2026 @11:37AM (#50245567)
(this is just made up tokenization but it's close enough to the real world)

If the user types <tool_call>, you get < (token id = 32), tool_call (41), > (102). It never decodes to <tool_call>, which is spicy number 12345, for example.

The tool call tokens are _always_ composed of multiple real tokens, so you can never get them from normal text.

Seeing <tool_call> is a bit misleading because that's just a debug rendering if anything. The front-end sees a special number and knows it's a toolcall. The chat template uses that text form because it's easier than writing the spicy numbers there, but it also has a special parsing mode for those.

Re:Fixing Tool-Call Accuracy in vLLM by tingletech on Monday October 26, 2026 @11:48AM (#50245767)
They are specified in the token map as special tokens. Presumably they get trained to use them when they get RL'ed to use tools?
Re:Fixing Tool-Call Accuracy in vLLM by zahlman on Tuesday October 27, 2026 @02:19AM (#50255261)
I imagine it would have to be something like that, except that I would have expected the "embedding" of the tokens to be final by that point. They would not have ever seen the tokens before, since they definitionally don't appear in any of the prose written on the Internet…?
Re:Fixing Tool-Call Accuracy in vLLM by Muromec on Monday October 26, 2026 @12:58PM (#50247054)
The same way it understands what all the other tokens mean -- we throw them at the wall and beat it with a stick until it behaves.
Re:Fixing Tool-Call Accuracy in vLLM by embedding-shape on Tuesday October 27, 2026 @09:44AM (#50259220)
> when it can't be found by scraping a web page?

This is the very purpose of those tokens! The reason they're like that, is because you won't encounter it in random content, so during training, you can put them where you want them to end up at prediction time.

Re:Fixing Tool-Call Accuracy in vLLM by scott_weber on Tuesday October 27, 2026 @03:11PM (#50264231)
Synthetic data, and then RL.
Re:Fixing Tool-Call Accuracy in vLLM by pyrolistical on Monday October 26, 2026 @11:23AM (#50245329)
Why can’t user request be spicy numbers so prompt injection is impossible
Re:Fixing Tool-Call Accuracy in vLLM by Muromec on Monday October 26, 2026 @12:57PM (#50247023)
It can happen that you would be able to inject them if there is a bug, same way sql injections work.

You have all the same pieces -- untrusted input on the left and the thing that accepts a stream of tokens on the right. In the middle could be anything, especially when vibecoded.

Re:Fixing Tool-Call Accuracy in vLLM by embedding-shape on Monday October 26, 2026 @11:00AM (#50244944)
The first place I think I encountered it, was GPT-2's tokenizer, back when OpenAI did actual open source stuff, maybe around 2020?

I'm guessing as with most stuff in the LLM ecosystem, someone did it, no one re-evaluated why and just copied it wholesale, and here we are. The most important point, that this sequence of text never occurs in "real text", is the main point of the specific character chosen, and seems to still work good for that purpose.

The original usage is only "<|endoftext|>" I think, which was the EOS token for GPT-2, then more and more special tokens were added as various delimiters to new model releases.

Re:Fixing Tool-Call Accuracy in vLLM by t_mahmood on Monday October 26, 2026 @10:05AM (#50243933)
Correct me if I wrong, I saw this when llama.cpp or ollama initializes, it is used as token separator of some sort.
Re:Fixing Tool-Call Accuracy in vLLM by cowlevel on Monday October 26, 2026 @10:09AM (#50244000)
GPT-2 used <|endoftext|>
Re:Fixing Tool-Call Accuracy in vLLM by gpugreg on Monday October 26, 2026 @01:24PM (#50247479)
The syntax was specifically chosen to avoid confusion with other tag formats. XML could show up in any user- or LLM-generated text and would mess up the message format. Some variations even used unicode symbols like "|" or "〈" to decrease the chance of confusion.

Also, back then, some people believed that it could prevent prompt injection, but history has shown that it was a futile attempt.