News for hackers. Stuff that matters.
Re:Show HN: LLMs can generate valid JSON 100% of the time by RyEgswuCsn on Monday August 14, 2023 @08:29PM (#37128603)
How does the LLM know what valid JSON tokens are?

What if the training data contains malformed JSON? There ought to be a non-zero chance of the LLM producing invalid JSON, no?

Re:Show HN: LLMs can generate valid JSON 100% of the time by catlifeonmars on Monday August 14, 2023 @08:40PM (#37128690)
* * *
Re:Show HN: LLMs can generate valid JSON 100% of the time by MereInterest on Monday August 14, 2023 @08:44PM (#37128720)
LLMs work by outputting a value for each token, then using those values to generate a probability distribution. Usually, this will be through a function like softmax [0], but there's nothing preventing you from doing some post-processing first. That post processing could be aware of the tokens that would be valid as the next token in a JSON format, and set the probabilities of all other tokens to zero. That way, even if the training data contains malformed JSON, the generator is still constrained to produce valid JSON.

[0] https://en.wikipedia.org/wiki/Softmax_function

Re:Show HN: LLMs can generate valid JSON 100% of the time by RyEgswuCsn on Wednesday August 16, 2023 @08:28AM (#37146041)
But that is not the LLM learning to produce valid JSON —— as some other commenters mentioned, you can get valid json without the LLM.

Sure it could be useful, but not really impressive.