This makes no sense. You can describe the brain reductively enough and make it sound like it can't have an original insight either. Transformers are expressive enough function approximaters in theory, there's no reason why a future one couldn't have novel insights.

This is such a weird misconception I keep seeing - the fact that the loss function during training is minimising CE/maximizing prob of correct token doesn't mean that it can't do "real" thinking. If circuitry doing "real" thinking is the best solution found by SGD then it obviously will

you're conflating LLMs with AI. Strictly speaking, from what we understand of physics, chemistry, biology, computing, and mathematics, there is no coherent argument that you could not build an AI which could be an effective research scientist (which I define as: a system which produces novel hypotheses based on current scientific knowledge that are likely to be true, and is capable of evaluating their likelihood quickly enough to be relevant to human endeavors.)

I imagine that such a system would probably have at least required component that looked much like an LLM.

Not all breakthroughs happened due to original insight- many came from tediously improving techniques through fairly mundane means, or from advancements in other areas.

> which I define as: a system which produces novel hypotheses based on current scientific knowledge that are likely to be true, and is capable of evaluating their likelihood quickly enough to be relevant to human endeavors.

Produces hypotheses which are likely to be true? Pardon my ignorance, have we even proven gravity to be true yet? Sure, I think gravity exists and is true, however your definition of AI seems like Swiss cheese.

I think you must have misunderstood my statement on multiple levels (or you're being disingenuous; it's hard to tell). I don't think you can ever prove anything to be true in science, and I expressed that in my statement ("likely to be true"). You could also turn what I said around, and say "novel hypotheses which are unlikely to be trivially falsified".

There's nothing swiss cheese about my heuristic definition of a research scientist AI; it's precisely the same thing that we expect human research scientists to do (I presume an AI research scientist could also write papers that get published and grants that get approved, but unlikely to be an effective mentor for a PhD candidate). It's also only a working definition that I would update if I found a good reason.

Sorry, I did not mean to be disingenuous. My skepticism came through in my tone. I think LLMs are a valuable tool, and will continue to grow in different ways. I can even imagine a scenario where the right series of prompts gives someone a eureka moment in a big, new way.

I don’t see how an LLM could produce anything novel without being prompted. At which point, is the LLM just a tool for a brilliant mind, or is the LLM the brilliant mind? I prefer the former, because I just can’t wrap my head around the latter.

Why do you keep talking about LLMs? An AI research scientist would almost certainly contain many more components.

Because it’s not AI.

I wouldn’t be so dismissive. Research is just a loop of hypothesis, experiments, collect data, make new hypothesis. There’s so creativity required for scientific breakthroughs, but 99.9% percent of scientists don’t need this creativity. Just need grit and stamina.

I wouldn't be so dismissive of the objection.

That loop involves way more flexible goal oriented attention, more intrinsic/implicit understanding of plausible cause and effect based on context, and more novel idea creation than it seems.

You can only brute force things with combinatorics and probabilities that have been well mapped via human attention, as piggy-backing off of lots of human digested data is just a clever way of avoiding those issues. Research is by definition novel human attention directed at a given area, so it can't benefit from that strategy in the same way domains which have already had a lot of human attention can.

I think the whole idea of "original insight" is doing a lot of heavy lifting here.

Most innovative is derivative, either from observation or cross application. People aren't sitting in isolation chambers their whole lives and coming up with things in the absence of input.

I don't know why people think a model would have to manifest a theory absence of input.

> I think the whole idea of "original insight" is doing a lot of heavy lifting here.

This is by biggest issue with AI conversations. Terms like "original insight" are just not rigorous enough to have a meaningful discussion about. Any example an LLM produces can be said to be not original enough and conversely you could imagine trivial types of originality that simple algorithms could simulate (i.e. speculate on which existing drugs could be used to treat known conditions). Given the amount of drugs and conditions you are bound to propose some original combination.

People usually end up just talking past each other.

And insight. Insight can be gleaned from a comprehensive knowledge of all previous trials and the pattern that emerges. But the big insights can also be simple random attempts people make because they dont know something is impossible. While AI _may_ be capable of the first type, it certainly won't be capable of the second

I think this comment is significantly more dismissive of science and scientists than the original comment was of AI.

Awfully bold to claim that 99.9% of scientists lack the need for "creativity". Creativity in methodology creates gigantic leaps away from reliance on grit and stamina.

And why is there even a desire to replace research scientists? Presumably this is the kind of job humans find meaningful and are good at. I don't understand AI as a replacement for humans instead of a smart tool for humans to make use of.

Why is there even a desire to replace software developers? Presumably this is the kind of job humans find meaningful

Why is there even a desire to replace car manufacturers? Presumably this is the kind of job humans find meaningful

[...]

Reduce labor costs to increase profits. But that's for the good of a small number of people when all meaningful work can be automated. And I don't trust the billionaires to make society better for the rest of us.

Why? to increase productivity and improve the human condition. If AI can do research then technological and scientific progress will increase dramatically.

In a world where human labor is no longer necessary, why do you think the people who control the AI models will care about “improving the condition” of anyone but themselves?

Human labor hasn't been necessary for a large portion of people. We have more than enough people to make food and collect water for us to survive. We can always make up jobs for people to do.

There's an unknown cost if all human endeavor becomes replaceable by AI. I would be cautious about this.

For who though? The research scientist can no longer do the work they trained for. Apply this to the entire economy at some point, and you have a serious problem.

Talk to the person who pays the scientist. Anyone who works, works for someone. Who you work for is the person who wants to replace you.

I don't see that as a net good for society. I do see it as eventual grounds for another French-style Revolution.

Or Luddites destroying machines that replace them while using machines as tools to make their labor easier.

The working class wants to automate labor to increase their free time.

The owner class wants to automate labor so they can rid themselves of the working class.

Society doesn't work because everyone is collaborating for the greater good. Society works because everyone generally cares very little about society and they care more about themselves. The net effect of everyone caring only about themselves is the result you have today.

That's a very western, libertarian way of looking at things. But even so, replacing workers en masse leads to high unemployment and civil unrest. If we get to the point of automating research scientists, then a whole lot of jobs can be automated.

Most of society and people on HN are from western society. I'd say it's more of a capitalist view point. Which is basically all of society you see today.

To ensure net benefit to society one must make policies that benefit individuals and hope that the network effects balance out and make society better overall. Expecting altruism is unrealistic.

The biggest problem with LLMs isn't that it lacks original insight. It's that the insight is so original that we call that insight hallucinations.

We like to think Humans are the most creative things on the face of the earth and we don't like to attribute creativity to LLMs. The sad reality is that LLMs are likely more creative then humans.

I think the distinction is that hallucinations are incorrect. You can be super creative building a new chair, but if you can’t sit in it, it’s not a chair.

Right. So you have a testing framework/agent/other llm. It’s not like our brain is one independent machine. It’s various parts all contributing different aspects of intelligence.

Most humans are also too creative, but we have moderating impulses that tell us so much. Very few humans have the skill of being able to ride the cutting edge without going too far off either side of it, and most can only do that in a very narrow subfield.