That's not really true any more, since given tool use and iterative 'chain of thought' reasoning, it's more like an answer refined by conversation with the asker or the second best answerer, from the best answerer, is the best that you can expect.

E.g. consider the best answer describes how to achieve something with some CLI, but the CLI's changed and the command is now broken. It can discover that, combine with the 'best answer' about what changed in the CLI, and fix it to work for the original problem even if that's completely novel and there were no extant answers about solving the problem with the new CLI form, or that it had broken that use case.

It doesn't change fundamentally how they work of course, but 'thinking', creating that feedback loop, made them significantly more useful IMO.

The “conversation” is just one between some redditors or book publishers.

It’s going to narrow in on a statistically better answer but it’s not going to create novel insight beyond the arguments made from compressed snapshots of existing knowledge.

There is no “thinking”. Just more attention, which is apparently, all you need.

That's not what "attention" means.

> it’s not going to create novel insight beyond the arguments made from compressed snapshots of existing knowledge

So all of the maths breakthroughs were not novel?

Firstly, many of them are still being checked, so hold your horses.

What I’ve found LLMs to be OK at when it comes to novel insight is being given a goal to work towards (“prove or disprove this”), with access to a large body of sources and it can then chase down 1000s of options no human would bother with.

This is why security and maths proofs seem to be an area where it can accelerate things. However I know from my own work in these areas it can still hallucinate and so the output needs careful checking.

Is that novel ideation? Or is it self-organising grid search? It’s more efficient than getting 1000 mathematicians to do the work, but it’s not doing something “new”, it’s building on existing work.

And yes, I know what “attention” means, I was making a pun. And it still stands up: they’re not in some deep thought about this and reflecting on new ideas, they’re just hammering away at everything in the literature.

And it still might be bullshit.

All novel insights expressible as text can be assembled from snippets of existing text. The challenge is "only" that you need to find the right snippets. Hence being a stochastic parrot does not represent an upper bound on capabilities, but a suggestion that if you want your parrot to give you a specific answer, you should arrange for a context where it's statistically likely to be correct.

[delayed]

No. In a large enough state space, you can of course easily find a state that has never existed before. For example, if you pick a 1024-bit integer uniformly at random, it will certainly be different from any previous 1024-bit integer. However, it's much more difficult to find a new integer that cannot be assembled out of bits taken from previous integers. Composition from a handful of base atoms can cover very large state spaces, and unbounded composition can cover infinite state spaces. Which is why only being able to combine things that already exist is not much of a limitation.

To demonstrate:

"Are you saying that a" https://news.ycombinator.com/item?id=49642931

"complex stateful system" https://news.ycombinator.com/item?id=14881439

"cannot generate a" https://news.ycombinator.com/item?id=45956162

"state that is not already" https://news.ycombinator.com/item?id=43013637

"recorded in text" https://news.ycombinator.com/item?id=44731577

"somewhere?" https://news.ycombinator.com/item?id=50391679

A novel state, obtainable by combining six existing ones.

>It’s going to narrow in on a statistically better answer but it’s not going to create novel insight beyond the arguments made from compressed snapshots of existing knowledge.

ignoring the fundamentally dishonest representation of LLMs in this subthread..

if humanity was given a tool that could restructure already existing and known knowledge and facts into novel results with a statistically better answer automatically it wouldn't be as a minor a win as you make it sound; it would be tremendous.

yeah chain of thought reasoning isnt really a think besides running the model for longer and hide most of the output (but still charging you for it). it literally does not work any differently than "non-reasoning" models (except for marketing)

> The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.

Hard disagree.

Claude is able to come up with solutions in very niche areas where there is little discussion on the web, because it takes steps to digest the source of (for instance) frameworks and libraries, and can then use/advise on lots of esoteric options, edge-cases and all sorts of stuff that doesn't get discussed on the open web.

I work in a niche where it is utterly hopeless at this, and everyone in the niche agrees: it produces things that look convincing to non-experts but that is fundamentally derivative dross.

It’s great at bread and butter stuff, but not at specialised ideation.

If you know differently, please share examples as maybe we can learn something here.

> It’s great at bread and butter stuff, but not at specialised ideation.

This is fundamentally different from your earlier, stronger claim that - "The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago."

I use a less popular java framework quite a lot, Micronaut (I want to move away from it, but that's a different discussion).

When trying to solve problems with its behaviour, I can watch as Claude digests the framework source to come up with suggestions, identify bugs that are causing behaviour regressions, and generally unlock stuff that I've previously found very frustrating. Frustrating because the documentation covers a tiny amount of what's in there, and frustrating because there's little online discourse about any of it to refer to.

Is this earth shattering in itself? Probably not. But it's certainly better than you can find from "an online forum like Reddit or a book/paper published more than a year ago".

You're the one making the claim, and also claiming that you have specific examples to back up your point. So maybe you should share them, and we can see if your assertions hold up.

You’re wrong on literally every point.

Nobody solves the Quasi-Riemann Hypothesis by looking up online forums.

The discussion here is talking about helping individual research with proposals a human will read. That is different from spending 10s of millions to do a brute-force search of lots of different proofs which can be verified by LEAN automatically. There is no "automatic verification" of a system design proposal, and no way to figure out from 10s of millions of proposals which is best automatically, and also assuming people in this discussion don't have 10s of millions of dollars to throw at every design problem.

> That is different from spending 10s of millions to do a brute-force search of lots of different proofs which can be verified by LEAN automatically

It’s kinda like the project where someone generated a lot of midi files for all possible pop riffs to copyright them. If you have a start and endpoint, rules for navigation, and unlimited computes, you can brute force anything. The nice stuff is when you made exploration efficient.

Other than for maybe Navier-Stokes, they just didn't brute force it in the way you guys are thinking. Most unsolved math problems are not susceptible to brute force, which is why the problems remained unsolved. AI is so quick it can of course try more ideas than a person, but each idea is executed at genius IQ level. Even if you knew the broad outlines of the proof of quasi-Riemann, there are probably only 20 people in the world who could have executed it.

This is trivially true for any theorem with a proof that can be automatically verified in finite time, but does not preclude the universe lacking a suitable combination of monkeys, typewriters, and time to do so.

>That is different from spending 10s of millions to do a brute-force search of lots of different proofs which can be verified by LEAN automatically.

That is not remotely how the vast majority of the proofs have worked. You don't know what you're talking about, but that seems par the course for these discussions.

I guarantee that you could not find a proof using brute force, even with literally trillions of dollars of compute. You're betraying a lack of understanding of the complexities involved, unless you're using a definition of "brute force" which is essentially equivalent to "intelligent search."

Brute force is certainly the wrong way to put it, but that doesn't mean it's easy to distinguish superhuman effort from superhuman intelligence, or, in the words of Thomas Edison, perspiration from inspiration.

Especially as the pure gruntwork involved in acquiring the knowledge required to merely understand the details of the mathematics involved represent at least a significant fraction of a human lifetime.

And unless this human process of knowledge acquisition continues, eventually no one will be left to understand and attach any significance to the results machines produce.

They do by reading papers and connecting dots.

Which is what they claim to have done.

>The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.

The LLMs are vastly superior for any subject matter with even the slightest hint of danger because while both LLMs and Reddit will whine at you and talk down to you the LLM will actually knock it off when told.