News for hackers. Stuff that matters.
Re:Structured Outputs in LLMs by dcreater on Tuesday September 23, 2025 @11:20AM (#45348322)
How does this compare to pydantic ai?

I'm yet to see a thorough comparison of design, performance and reliability between these options (along with outlines etc)

Re:Structured Outputs in LLMs by FlyingLawnmower on Tuesday September 23, 2025 @01:38PM (#45350321)
We did quite a thorough benchmarking of various structured decoding providers in one of our papers: https://arxiv.org/abs/2501.10868v3 , measuring structured outputs providers on performance, constraint flexibility, downstream task accuracy, etc.

Happy to chat more about the benchmark. Note that these are a bit out of date though, I'm sure many of the providers we tested have made improvements (and some have switched to wholesale using llguidance as a backend)

Re:Structured Outputs in LLMs by 0x4FFC8F on Tuesday September 23, 2025 @03:13PM (#45351454)
I think @dcreater was asking how these various structee decoding providers compare with how pydantic ai handles structured output, i.e via tool calling, forcing the LLM to use a tool and its arguments are a json schema hence you read the tool call arguments and get a structured output.
Re:Structured Outputs in LLMs by dcreater on Thursday September 25, 2025 @12:23AM (#45369254)
thanks for the paper link! Im surprised there is such a minimal improvement in structured outputs when using any of these tools over the bare LLM!
Re:Structured Outputs in LLMs by _andrei_ on Tuesday September 23, 2025 @01:57PM (#45350572)
pydantic is a _validation_ library, it does not do any kind of constraints by itself
Re:Structured Outputs in LLMs by dcreater on Tuesday September 23, 2025 @03:25PM (#45351573)
im referring to pydanticai https://ai.pydantic.dev/