He's speaking about his objective to make ever stronger LLMs: so for this his secondary objective is to measure their real performance.

The human preference is not that good of a proxy measurement: for instance, it can be gamed by making the model more assertive, causing the human error-spotting ability to decrease a lot [0].

So what he's really saying is that non-rigorous human vibe checks (like those LMSys Chatbot Arena is built on, although I love it) won't cut it anymore to evaluate models, because now models are past that point. Just like you can't evaluate how smart a smart person really is in a 2min casual conversation.

[0]: https://openreview.net/pdf?id=7W3GLNImfS

ie when you cant beat them, make new metrics

and you can absolutely evaluate how smart someone is in a 2min casual conversation. You wont be able to tell how well they are in some niche topic, but %insert something about different flavors of intelligence and how they do not equate do subject matter expertise%

It’s a common pattern that AI benchmarks get too easy, so they make new ones that are harder.

As models improve, human preference will become worse as a proxy measurement (e.g. as model capabilities surpass the human's ability to judge correctness at a glance). This can be due to more raw capability - or more persuasion / charisma.

It's trivial to come up with prompts that 4o fails. If it's hard to come up with prompts that 1o succeeds on but 4o fails, that implies the delta is not that great.

Or, the delta depends on the nature of the problem/prompt, we’ve not yet figured that out, there’s a relatively narrow range of prompts with large delta, and so finding those examples is a work in progress?

> Results are "strong" but can't be felt by the user? What does that even mean?

Not every conversation you have with a PhD will make it obvious that that person is a PhD. Someone can be really smart, but if you don't see them in a setting where they can express it, then you'll have no way of fully assessing their intelligence. Similarly, if you only use OAI models with low-demand prompts, you may not be able to tell the difference between a good model and a great one.

> Results are "strong" but can't be felt by the user? What does that even mean?

This means it often doesn't provide the answer the user is looking for. In my opinion, it's an alignment problem, people are very presumptuous and leave out a lot of detail in their request. Like the "which is bigger - 9.8 or 9.11? question, if you ask "numerically which is bigger - 9.8 or 9.11?" It gets the correct answer, basically it prioritizes a different meaning for bigger.

> But the last sentence is the worst: "we all need to find harder prompts". If I understand it correctly, it means we should go looking for new problems / craft specific questions that would let these new models shine. But why? Why would we do that? Wouldn't our time be better spent trying to solve our actual, current problems, using any tool available?

Without better questions we can't test and prove that it is getting more intelligent or is just wrong. If it is more intelligent than us it it might provide answers that don't make sense to us but are actually clever, 4d chess as they say. Again an alignment problem, better questions aid with solving that.

The irony here is that Jason is speaking in the context of LLM development, which he lives and breaths all day.

Reading his comments without framing it in that context makes it come off pretty badly - humans failing to understand what is being said because they don't have context.

I think they are saying they have invented the screwdriver. We have all been using, hammers to sink screws, but if you try this new tool it may be better. However, you will still encounter a lot of nails.

It's more like they're saying they have invented the screwdriver, but they haven't invented screws yet.

But it doesn't feel right. It's unlikely the screwdriver would come first, and then people would go around looking for things to use it with, no?

It's more like they have invented a computer, an extremely versatile and powerful tool that can be used in many ways, but is not a solution to every problem.

Now they need people to write software that uses this capability to perform useful tasks, such as text processing, working with spreadsheets and providing new ways of communication.

While I find value in LLMs they still overall seem unreasonably not that useful.

It might be like trying to train a neural net in 1993 on a 60mhz Pentium. It is the right idea but fundamental parts of the system are so lacking.

On the other hand, I worry we have gone down the support vector machine path again. A huge amount of brain power spent on a somewhat dead end that just fits the current hardware better than what we will actually use in the long run.

The big difference though from SVM is this has captured the popular imagination and if the tide goes out, the AI winter will the most brutal winter by an order of magnitude.

AGI or bust.

I’d say the biggest difference between LLMs and SVMs is that a lot of people find LLMs useful on a daily basis.

I’ve been using them almost daily for over two years now, and I keep on finding new things they can do that are useful to me.

Is there a post on your blog that lists your different uses of LLMs?

Not in a single place, but it came up in a podcast episode the other day - about 32 minutes in to this one I think https://softwaremisadventures.com/p/simon-willison-llm-weird...

They’re useful, but not for what AI companies seem to be pushing for.

I like that they can reorganize my data, document QA is pretty killer as long as the document was prepared well.

Embeddings are sick.

But content creation… not useful. Problem solving? Personally have not found them useful (haven’t tried o1 yet)

> What does that even mean?

It explicitly says "Results on AIME and GPQA are really strong". So I would assume it means it can get (statistically significantly, I assume) better score in AIME and GPQA benchmarks compared to 4o.

> But why? Why would we do that?

Because OpenAI needs a steady influx of money, big money. In order to do so, they have to convince the people who are giving them money that they are the best. An objective way to achieve this is by benchmarking. But once you enter this game, you start optimizing for benchmarks.

At the same time, in the real world, Anthropic is following them in huge leaps and for many users Claude 3.5 is already the default tool for daily work.

Agree completely.

From a user perspective too, I was a subscriber from the first day of gpt4 until about a month ago. I thought about subscribing for the month to check this out but I am tired of the OpenAI experience.

Where is Sora? Where is the version of chatgpt that responds in real time to your voice? Remember the gpt4 demo that you would draw a website on a napkin?

How about Q* lol. Strawberry/Q*/o1, "it is super dangerous, be very careful!"

Quietly, Anthropic has just kicked their ass without all the hype and I am about to go work in sonnet instead even bothering to check o1 out.

This is something we've been grappeling with on my team. Many of the researchers in the org want to try all these reasoning techniques to increase performance, and my team keeps pushing back that we don't actually need that extra performance- we just want to decrease latency and cost.

So make the requirement using a cheaper and lower latency model and try to increase the performance to a satisfactory level. Assuming that you are not already using the cheapest/lowest latency model.

You're reading too much into an offhand comment that's more metaphorical in nature.

Dont you know by now

Speaking with AI maxis it’s easy:

The AI is always right

You are always wrong

If AI might enable something dangerous, it was already possible by hand, scale is irrelevant

But also AI enables many amazing things not previously possible, at scale

If you don’t get the answers you want, you’re prompting it wrong. You need to work harder to show how much better the AI is. But definitely, it cannot make things worse at scale in any way. And anyone who wants regulations to even require attribution and labeling, is a dangerous luddite depriving humanity of innovations.

The stupidest thing about ai and automation is that they are trying to target it at large corporations looking to cut down on jobs or 10x productivity when all anyone actually wants is a robot to do their laundry and dishes.

You're describing a dish-washer and washing-machine.

The GP is almost certainly describing a robot that can move dirty stuff into the machines, run them, and put away the clean stuff afterwards.

these are almost entirely unrelated problems

Because a robot that do everyone's laundry is much more closer to AGI than ChatGPT. I'm dead serious.

Not really. You don't need to move wet clothes from the first machine to a second machine if you get one machine that does both jobs. That's very much not AGI. The second job, of taking dry crumpled clothes and folding them, also doesn't need an artificial general intelligence. It's very computationally expensive (as evidenced by the speed of https://pantor.github.io/speedfolding/, out of UC Berkeley) and a hard robotics question, but it's also very fixed function.

Taking the clothes out of the combined washer dryer machine, my laundry folding robot isn't suddenly going to need to come up with a creative answer to a question I have about politics in order to fold the laundry, or come up with a new way to organize my board game collection, or reason about how to refactor some code. There are no logical leaps of reasoning or deep thinking required. My laundry folding robot doesn't need to be creative in order to fold laundry, just application of some very complex algorithms, some of which have yet to be discovered.

This hits the nail on the head. It is a consumer facing product not a technology to solve deep thinking.

i don't think that's what he's saying

> we all need to find harder prompts

"One of the biggest traps for engineers is optimizing a thing that shouldn't exist." (from Musk I believe)