Show HN: LLMs can generate valid JSON 100% of the time

Re:Show HN: LLMs can generate valid JSON 100% of the time by IanCal on Tuesday August 15, 2023 @08:02PM (#37140999)

I tried slight modifications from the example pydantic model and it's incredibly slow. Maybe I'm doing something wrong but I've a hefty box and a 3090, an example using gpt-2 doesn't seem like it should be that taxing.

[ Parent | on HN ]

Re:Show HN: LLMs can generate valid JSON 100% of the time by remilouf on Thursday August 17, 2023 @05:47PM (#37168594)

It is currently limited by the time it takes to build the index. There are obvious optimizations we can apply to this, however in a production setting it does not matter much since you only need to build the index once for each (schema, vocabulary) pair.

[ Parent | on HN ]

Re:Show HN: LLMs can generate valid JSON 100% of the time by IanCal on Saturday August 19, 2023 @05:09AM (#37186935)

Is there a rough guide as to how long to wait? I think it's definitely an important thing if building takes 10+ minutes (or hours?) for even very basic models, that's a fundamentally different production architecture (as launching from a blank slate is now not feasible). It's also a big devx issue.

I'd highlight this somewhere on the readme as I wasn't sure if it was just broken or how long to wait.

[ Parent | on HN ]