News for hackers. Stuff that matters.
Re:Show HN: LLMs can generate valid JSON 100% of the time by quickthrower2 on Monday August 14, 2023 @06:55PM (#37127827)
> LLMs can generate valid JSON 100% of the time

If that seems surprising, it is worth doing a course like Karpathy's zero to hero NN, and have all the magic peeled away a layer at a time.

The reason you can do this is because LLMs don't just generate the next word or token, it produces a probability distribution over all tokens. A JSON parser can give you a list of next valid tokens. The tokens in each case might be from a different set, e.g LLM thinks of " The" whereas the JSON parser might think of "{", so you need some conversion there. But if you sample randomly from only the valid tokens, the output must be valid JSON.

What you can't build a parser for though is ... the truth! You may still be told lies or made up stuff.

Re:Show HN: LLMs can generate valid JSON 100% of the time by dwattttt on Monday August 14, 2023 @07:41PM (#37128262)
If you're choosing the next token based on a list of valid next tokens, a uniform random distribution can always generate valid JSON too!
Re:Show HN: LLMs can generate valid JSON 100% of the time by quickthrower2 on Monday August 14, 2023 @07:54PM (#37128365)
Yep. So can this:

    fun generate_valid_json(seed):
        return "{}"
Re:Show HN: LLMs can generate valid JSON 100% of the time by neoncontrails on Monday August 14, 2023 @09:04PM (#37128874)
But that's not what an LLM does.
Re:Show HN: LLMs can generate valid JSON 100% of the time by antonvs on Monday August 14, 2023 @09:55PM (#37129229)
The point is that if you're "choosing the next token based on a list of valid next tokens," it's not surprising that you'll only generate valid output, since absolutely any choice mechanism will suffice.
Re:Show HN: LLMs can generate valid JSON 100% of the time by mattigames on Monday August 14, 2023 @07:21PM (#37128113)
It's not like humans are particularly good at distinguishing truth from lies.
Re:Show HN: LLMs can generate valid JSON 100% of the time by quickthrower2 on Monday August 14, 2023 @07:33PM (#37128203)
The word "lie" is probably too anthropic here. I should have just said "made up". There is no intent to lie. And the model isn't try to self-fact-check anyway. (Maybe some do). But if they do they are probably bad at it at the moment, at least from my experience of GPT3.5 (not used 4 much).
Re:Show HN: LLMs can generate valid JSON 100% of the time by coder543 on Monday August 14, 2023 @07:58PM (#37128396)
> at least from my experience of GPT3.5 (not used 4 much).

And 4 is tremendously better than 3.5, in my own experience. Not perfect, but actually useful.

Re:Show HN: LLMs can generate valid JSON 100% of the time by quickthrower2 on Monday August 14, 2023 @08:25PM (#37128586)
Can anyone recommend a good, and trusted UI so I can use it via the API? I don't want to pay monthly for it, but would be nice to use occasionally. I keep meaning to do this!
Re:Show HN: LLMs can generate valid JSON 100% of the time by selcuka on Monday August 14, 2023 @09:23PM (#37129024)
OpenAI has its own playground where you can test all models (I believe GPT-4 is not available to everyone yet):

https://platform.openai.com/playground

Monthly subscription is only for ChatGPT. When you use the APIs you pay per token.

Re:Show HN: LLMs can generate valid JSON 100% of the time by coder543 on Monday August 14, 2023 @11:25PM (#37129803)
> I believe GPT-4 is not available to everyone yet

I still don't have access, except through the regular ChatGPT interface, which is mildly annoying. It would be interesting to experiment with the API.

Re:Show HN: LLMs can generate valid JSON 100% of the time by dilawar on Monday August 14, 2023 @10:08PM (#37129324)
"Its a human nature to mislead others, sometimes knowingly." I read this line in an anthropology book. A similarly non-cynical approach towards your fellow is "trust but verify".
Re:Show HN: LLMs can generate valid JSON 100% of the time by RyEgswuCsn on Monday August 14, 2023 @08:29PM (#37128603)
How does the LLM know what valid JSON tokens are?

What if the training data contains malformed JSON? There ought to be a non-zero chance of the LLM producing invalid JSON, no?

Re:Show HN: LLMs can generate valid JSON 100% of the time by catlifeonmars on Monday August 14, 2023 @08:40PM (#37128690)
* * *
Re:Show HN: LLMs can generate valid JSON 100% of the time by MereInterest on Monday August 14, 2023 @08:44PM (#37128720)
LLMs work by outputting a value for each token, then using those values to generate a probability distribution. Usually, this will be through a function like softmax [0], but there's nothing preventing you from doing some post-processing first. That post processing could be aware of the tokens that would be valid as the next token in a JSON format, and set the probabilities of all other tokens to zero. That way, even if the training data contains malformed JSON, the generator is still constrained to produce valid JSON.

[0] https://en.wikipedia.org/wiki/Softmax_function

Re:Show HN: LLMs can generate valid JSON 100% of the time by RyEgswuCsn on Wednesday August 16, 2023 @08:28AM (#37146041)
But that is not the LLM learning to produce valid JSON —— as some other commenters mentioned, you can get valid json without the LLM.

Sure it could be useful, but not really impressive.

Re:Show HN: LLMs can generate valid JSON 100% of the time by OJFord on Monday August 14, 2023 @08:40PM (#37128685)
Maybe it's just me, but I'm not doing anything that calls itself 'zero to hero'. Would love some good resources (preferably textbook, or at least written) on LLMs. I don't even understand the link to 'generative' image/video AI, which seems to have exploded at roughly the same time and surely isn't a coincidence.

I studied a little (literally 'intro to') ML at university, about enough to grok it as an application of stats, tie into things seen elsewhere, but not really more than that.

Every supposéd tutorial or explainer I've seen posted here or been able to find has been a weird (IMO) mix if simultaneously assuming a decent (at least greater than mine) ML background, but also really dumbed down clone this repo download that model switch between them like this, fine-tune them by cd'ing to this directory and ... Ok but what's actually going on?

Re:Show HN: LLMs can generate valid JSON 100% of the time by fenomas on Monday August 14, 2023 @09:26PM (#37129039)
The zero to hero video series is what you're looking for - look past the name and watch it. It's excellent.
Re:Show HN: LLMs can generate valid JSON 100% of the time by frontier on Monday August 14, 2023 @09:27PM (#37129045)
Karpathy's series is many many hours long and really does take you from zero to GPT. It's excellent! You sound triggered by the title - that may not even be the official title - but it definitely deserves it. Go look it up.
Re:Show HN: LLMs can generate valid JSON 100% of the time by OJFord on Monday August 14, 2023 @09:59PM (#37129257)
The title suggests I wouldn't like it, yes. But as a video series it's not 'a textbook or at least written' is i - not really the format I'm looking for personally.
Re:Show HN: LLMs can generate valid JSON 100% of the time by thatcherthorn on Monday August 14, 2023 @10:07PM (#37129309)
Truly.. one of the greatest minds in our ML era. Don't get caught up on the format :)
Re:Show HN: LLMs can generate valid JSON 100% of the time by OJFord on Tuesday August 15, 2023 @04:37AM (#37131554)
I just don't find it an effective way of learning personally. I didn't expect this to be so controversial - different people learn differently.
Re:Show HN: LLMs can generate valid JSON 100% of the time by quickthrower2 on Tuesday August 15, 2023 @06:51AM (#37132332)
I recommend this for groundwork to get you near LLM, and cover the journey deeply. I used some of this as a helper course for Karpathy. I learned things here he didn’t cover and vice versa. https://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/

I haven’t done tonnes of courses so there might be better. But this is good as a free one.

Re:Show HN: LLMs can generate valid JSON 100% of the time by OJFord on Wednesday August 16, 2023 @07:40AM (#37145554)
Thanks!
Re:Show HN: LLMs can generate valid JSON 100% of the time by frontier on Tuesday August 15, 2023 @06:28AM (#37132192)
Fair enough if you prefer to slog through an entire textbook. But for anyone else.. I can't recommend this series more highly, it was just amazing, no filler, pure step after step, explained methodically to the end goal.
Re:Show HN: LLMs can generate valid JSON 100% of the time by quickthrower2 on Tuesday August 15, 2023 @06:47AM (#37132304)
Lol I had to slog through some traditional material to keep up with Karpathy. A lot is covered in those videos.
Re:Show HN: LLMs can generate valid JSON 100% of the time by antonvs on Monday August 14, 2023 @09:57PM (#37129240)
> I'm not doing anything that calls itself 'zero to hero'.

Sounds like you have a case of the Mondays. You just need to turn that frown upside down!

Re:Show HN: LLMs can generate valid JSON 100% of the time by OJFord on Tuesday August 15, 2023 @04:59AM (#37131679)
Must have left my flair at home.
Re:Show HN: LLMs can generate valid JSON 100% of the time by pavs on Monday August 14, 2023 @10:18PM (#37129386)
Since you are "judging a book by its cover", or this case a name. This might interest you, that karpathy was co-founding developer of OpenAI, left to work at Tesla to head their AI development for 5-ish years and now back at OpenAI.

I can understand that you might be interested in book form only, I was lile this for the longest time, until I bumped into some really high quality video series that changed my mind to be a bit more flexible.

Also Important to note that LLM development is moving at a very fast pace recently so a book form might not be ideal. The basic ideas might be same, most of it might be out of date 6-12 months from now. I dont see how anyone could write a quality book that covers everything on this.

Re:Show HN: LLMs can generate valid JSON 100% of the time by OJFord on Tuesday August 15, 2023 @04:58AM (#37131677)
I realise that; also that it doesn't help that afaiui it's been more industry-led than academia.

But I truly am starting from pretty much 'zero', and maybe I wasn't clear but I'm not looking to be 'hero' in the sense of up to date with the cutting edge, or even necessarily putting anything in to practice at all, I'm more interested in the background theory, and fine with that missing the absolute latest extra technique, just want to understand the meat of it better.

A refresher on SVMs & PCA (which I barely remember - I think I could convincingly explain SVMs to someone numerate but non-tech/mathematician, but not otherwise) and then a catch up to roughly what's going on with LLMs & image/video as mentioned would be great.

> I can understand that you might be interested in book form only, I was lile this for the longest time, until I bumped into some really high quality video series that changed my mind to be a bit more flexible.

I enjoy videos for many things, but mostly entertainment, I don't personally find I can learn that well from them, especially more technical/theoretical stuff, sure to some combination of screen fatigue, it being harder to skip around and reference something, and distraction - something seems obvious briefly so my mind wonders, check something in another tab 'just quickly', and before you know it ten minutes have passed, I've been hearing the speaking but suddenly realise I haven't been listening, have no idea what's going on any more.

Obviously they work for some people, that's fine.

Re:Show HN: LLMs can generate valid JSON 100% of the time by quickthrower2 on Tuesday August 15, 2023 @06:53AM (#37132345)
Karpathy has flash attention of Pytorch in his repo. I understand that is fairly recent (in human weeks, maybe not AI dog weeks)