Every Way to Get Structured Output from LLMs

Re:Every Way to Get Structured Output from LLMs by hellovai on Tuesday June 18, 2024 @03:07AM (#40714901)

not a noob question, here's how the LLM works:

```

prompt = "..."

output = []

do:

  token_probabilities = call_model(prompt)

  best_token = pick_best(token_probabilities)

  if best_token == '<END>':

    break

  output += best_token

while true

return output

```

basically to support generation they would need to modify pick_best to support constraining. That would make it so they can't optimize the hot loop at their scales. They support super broad output constraints like JSON which apply to everyone, but that leads to other issues (things like chain-of-thought/reasoning perform way worse in structured responses).

[ Parent | on HN ]

Re:Every Way to Get Structured Output from LLMs by PheonixPharts on Tuesday June 18, 2024 @10:53AM (#40718510)

> things like chain-of-thought/reasoning perform way worse in structured responses

That is fairly well establish to be not true.

[ Parent | on HN ]