Show HN: LLMs can generate valid JSON 100% of the time

Re:Show HN: LLMs can generate valid JSON 100% of the time by anotherpaulg on Monday August 14, 2023 @06:58PM (#37127863)

For complex tasks like coding, my experience is that asking for a complex output format hurts performance on the underlying task. This showed up clearly in code editing benchmarks of GPT-3.5 and GPT-4:

https://aider.chat/docs/benchmarks.html

I’m curious if you have measured whether the “constrained generation” that you’re doing suffers from similar downsides?

[ Parent | on HN ]

Re:Show HN: LLMs can generate valid JSON 100% of the time by infecto on Monday August 14, 2023 @07:22PM (#37128117)

100% have observed the same over many tests. No loss in fidelity when responding in spoken language style of formatting but using json is disastrous.

[ Parent | on HN ]

Re:Show HN: LLMs can generate valid JSON 100% of the time by nouri on Monday August 14, 2023 @08:18PM (#37128531)

Using OpenAI Function Calls or asking for JSON in the prompt?

[ Parent | on HN ]

Re:Show HN: LLMs can generate valid JSON 100% of the time by infecto on Tuesday August 15, 2023 @06:58AM (#37132383)

I have noticed it in both but have been working with json output before function calling was introduced so I have more evidence on that side. The times I have tried to implement it in a function call I was equally unimpressed with it.

[ Parent | on HN ]

Re:Show HN: LLMs can generate valid JSON 100% of the time by speedgoose on Tuesday August 15, 2023 @03:03AM (#37131068)

While not ideal, could a workaround be to ask in spoken language first, and then ask to format it in JSON?

[ Parent | on HN ]

Re:Show HN: LLMs can generate valid JSON 100% of the time by infecto on Tuesday August 15, 2023 @06:56AM (#37132369)

That’s what we have been doing. Two passes. The task and then the format.

[ Parent | on HN ]

Re:Show HN: LLMs can generate valid JSON 100% of the time by darkteflon on Monday August 14, 2023 @07:48PM (#37128315)

We’ve seen this too. We run them as two separate stages - “reason”, log the intermediate output, then parse.

[ Parent | on HN ]