News for hackers. Stuff that matters.
Re:Gemini 4 Argon by TeMPOraL on Wednesday September 30, 2026 @04:57PM (#49914328)
> I think some in the AI industry drank their own Kool-Aid. They believed that if they had the best model and the most compute, they could tell the model, "Make a better model." And it would, and the next one could make its replacement, and so on.

They're not there yet. Once they get there, that's literally the definition of Singularity.

But they are getting closer. Recursive Self-Improvement used to be a phrase people mocked LessWrong crowd for using and worrying about, now it's something both OpenAI and Anthropic already publicly admitted not only to pursue, but to already be benefiting from.

Re:Gemini 4 Argon by SwellJoe on Wednesday September 30, 2026 @05:29PM (#49914725)
Sure, it's happening...but, is IT happening? By that, I mean, we can see that the models are able to iterate at a pace and scale that humans can't match, and that provides gains in model performance and efficiency. But, humans are still needed in the loop, and not just because it's necessary for safety/alignment reasons. I don't think any significant discovery has been made by models on their own, and I don't know that LLMs will ever have the capacity to invent. They can synthesize from known data amazingly well, and since they know everything "known data" is extremely broad. But, the leaps, so far, have all come from humans.

So far, I don't think the models are capable of running away on their own. Of course, it would be playing with fire to not at least consider the risks of such a runaway scenario and build in safeguards against it. But, there is no model that can build a better model on its own, thus far, to the best of my knowledge (which is far more limited than the models, so maybe I should ask them).

Re:Gemini 4 Argon by TeMPOraL on Wednesday September 30, 2026 @06:00PM (#49915064)
Recursive Self-Improvement isn't instant, it starts slow and accelerates.

It starts with what they already claim to be doing - increasingly relying on existing models in non-trivial work related to training, evaluating and optimizing the next, more capable generation of models. As long as the proportion of work keeps shifting towards agents doing more and more of it, and humans less and less, that's RSI at play.

It may be that it turns out LLMs lack some fundamental level of judgement and it plateaus, but frankly I find this notion absurd; LLMs already show better judgement than most people. The alternative is, at some point LLMs will show the ability to futz their way into improvement of the next generation of models even without humans in the loop - even if much less efficient at first, if generation N+1 is more capable than generation N, it'll either take off or burn out.

Re:Gemini 4 Argon by freecodeio on Wednesday September 30, 2026 @06:48PM (#49915472)
you are really just talking out of your ass here, no offense

just because more and more agents are doing human work, that in no way means the model somehow becomes magically more intelligent, it just means the work will stall and continue on at the same level forever

hell even if they hypothetically have an internal model that can output the entire training data set in a better format, there's no scientific evidence that the newer format has new information that is sufficient enough to train a better AI

as a matter of fact the scientific evidence is on the contrary

Re:Gemini 4 Argon by TeMPOraL on Wednesday September 30, 2026 @07:13PM (#49915684)
It's not the number of agents you should pay attention to, but the scope and nature of work they can perform effectively without being micromanaged by humans.
Re:Gemini 4 Argon by gytt33 on Wednesday September 30, 2026 @07:44PM (#49915930)
He talks out of his ass on a lot of topics on here especially those concerning economics and finance.
Re:Gemini 4 Argon by iamarobot on Thursday October 01, 2026 @11:13AM (#49922763)
Even currently, there are clearly diminishing returns throughout the field. While not a skeptic myself, I think the age of LLM's is now over or close to plateauing. GPT-6.1 Sol seems almost the same as GPT-6 Sol(even if the benchmarks are slightly better). It doesn't help that we are running out of real training data, requiring new Synthetic data to be created for the next generation of Models. AI(or SI), has hit its peak, IMO, and now the focus shifts directly to Autonomous tasks. While Fable 5.1 is borderline okay at autonomous tasks, the amount of Slop it generates fundamentally requires a human to correct its mistakes. I was working on a website for a nonprofit with Fable, but the amount of slop and corrections I had to make was astounding/
Re:Gemini 4 Argon by breuleux on Wednesday September 30, 2026 @11:36PM (#49917435)
> It may be that it turns out LLMs lack some fundamental level of judgement and it plateaus

All intelligence, LLM or not, is bound to plateau around the point where the need to operate within physical reality bottlenecks the speed of feedback. AI is progressing swiftly in the digital realm where feedback is nearly instantaneous, but it's unclear whether that would translate into improvements in the physical world where signals are much noisier and intelligence and judgment are less impactful.

Re:Gemini 4 Argon by TeMPOraL on Thursday October 01, 2026 @02:59PM (#49925695)
Our tried-and-true approach for when the physical world is causing us trouble, is to remake the physical world to be easier to work with.

My go-to example: wheels suck at mobility in the natural world. There's a reason no animal uses wheels as their means of locomotion. They only really work on flat, hard surfaces that give a good grip.

Our solution? We didn't build mechanical legs for all-terrain mobility. We paved the world instead. Dirt roads, then brick roads, then asphalt and concrete - since ancient history, we were forcing the world to adapt, so the one means of mobility we could easily built was effective.

We repeat this pattern every time when the world doesn't agree with us, and adapting ourselves to it is too much of a hassle.

Now, AI will likely adopt the same approach to these kinds of problems, which may not end up very nicely for us.

(Note: we are already adapting our own digital worlds to make them easier for AI. Much like building roads for our wheels, we now have a resurgence of CLI, with new tools exposing high-level operations optimized for agents - not humans - to use comfortably.)

Re:Gemini 4 Argon by breuleux on Sunday October 04, 2026 @11:13AM (#49954597)
Sure, but is the limiting factor intelligence, here? Give human intelligence to dolphins and they're still not going to be able to pave the world. In order to remake the physical world, we needed certain physical abilities -- we needed the animal mind and body that took billions of years to evolve and embeds advanced priors about the world, and we needed it packaged properly. Irrespective of our intelligence, our body is a general world navigation device adapted to grab and move stuff. We use this innate tool to remake the physical world, but we didn't make it, and we'd be impotent without it.

With AI it's a bit of a chicken or the egg situation. Before it can ablate the complexity of the world, it needs to be able to deal with it, including adversaries such as ourselves. For this it needs a tool as sophisticated as our bodies, but could AI develop such a tool for itself without the help of this tool? I'm not convinced.

Re:Gemini 4 Argon by pvab3 on Wednesday September 30, 2026 @05:29PM (#49914731)
Those people have a lot of overlap with the LessWrong crowd. They do not have RSI now and probably never will
Re:Gemini 4 Argon by TeMPOraL on Wednesday September 30, 2026 @05:55PM (#49915009)
They absolutely do, unless you believe they are lying about the fact they're using current generation models extensively to develop the next generation of their models.
Re:Gemini 4 Argon by Yizahi on Wednesday September 30, 2026 @06:19PM (#49915215)
The letter S stands for "Self" and word "using" is for sure not a superset of the word "self". Basically LLMs are assisting someone who does improving of said LLM, while RSI is a carpal... ahem, RSI is "self" improvement, meaning no intemediary in a human form. PS: it's also not recursive but iterative improvement, even if it ever happens.
Re:Gemini 4 Argon by TeMPOraL on Wednesday September 30, 2026 @07:06PM (#49915642)
At which point is the human using the model as a tool, and at which point is the model using human as a tool?

I can already see the border shift even for mundane tasks I have Claude working on. Increasingly, I'm just setting a high-level goal, and then checking progress and occasionally answering questions or doing something like configuring a system Claude can't easily reach itself (e.g. recording a bunch of traces through my normal use of a system that Claude deemed too fragile to risk operating on its own). Of course, I get detailed instructions to help me - "go there, do this and that, then press this to capture recording, run through this script here to process, attach result to next message". In those cases, Claude is effectively using me as a tool to call.

Re:Gemini 4 Argon by 8note on Wednesday September 30, 2026 @07:44PM (#49915929)
"improvement" is the word Id put the most emphasis on.

weve seen some improvement from the LLMs unattended, maybe, but will it actually keep improving vs needing a human to bring it back on track?

the recursive part is that it keeps improving on itself, but we really have no example of that. if it does it 30 times with improvements, then maybe, but even then, to actually be relevant it has to do better than paying scientists to do the work for the same cost, consistently.

RSI still means nothing if it costs 1000x the cost to get the same improvements as a human researcher

Re:Gemini 4 Argon by gytt33 on Wednesday September 30, 2026 @07:46PM (#49915950)
The physical constraint is money - which could be said anything. Could vehicle factories be more automated if we threw a gazillion dollars at it? Sure. Would it make economic sense? No. ROIC would be disatrous. No investor wants part of that.

This is the nuance that poster doesn’t understand. Given how much money thrown at it - we’re not even close. Who has the appetite to keep throwing more given they continually need to keep raising fresh money?

Re:Gemini 4 Argon by TeMPOraL on Thursday October 01, 2026 @03:00AM (#49918593)
> The physical constraint is money - which could be said anything.

Money is fake and not a constraint, that's literally the point of capital investments you guys are overindexing on so badly.

> Could vehicle factories be more automated if we threw a gazillion dollars at it?

They already are automated as much as it makes sense. Some of the automation is silicon-and-steel based, some of it is protein-based. Car manufacturers aren't in the business of pushing robotics and nanotechnology, so they prefer to hire protein automatons instead of developing and building their own, so yeah, they "pay people", but think what exactly they are paying them for.

Then consider that this is very much the work the AI is gradually getting as good as, or better, than us.

> This is the nuance that poster doesn’t understand. Given how much money thrown at it - we’re not even close.

What you seem to be missing is that "investing in AI" isn't investing in a chatbot, it's investing in technology that will (and already partially is) sit upstream of every other industry, of everything humans do. Like electricity or the Internet itself.

(The other thing you seem to not understand, in contrast with some of the investors, is that RSI is not a linear walk, it's an exponential curve. X-risk notwithstanding, by the time it's obvious to everyone, it's too late to make money investing in it.)

> keep throwing more given they continually need to keep raising fresh money

Have you heard about R&D?

I'm starting to think that "investors first" thinking that's so common here, that makes people feel they're smart, is actually quite backwards, especially at this scale. Or maybe it's simply people starting with a conclusion and trying to fit reality to match it, no matter how clear of a nonsense that conclusion is?

Re:Gemini 4 Argon by TeMPOraL on Thursday October 01, 2026 @03:26AM (#49918764)
> if it does it 30 times with improvements, then maybe, but even then, to actually be relevant it has to do better than paying scientists to do the work for the same cost, consistently.

Not necessary. Scientists are capacity limited and supply limited.

> RSI still means nothing if it costs 1000x the cost to get the same improvements as a human researcher

It means you can replace a human researcher with 1000x their salary burned on electricity. It also mean you can get two of them for 2000x of salary of one,

That's a bargain, actually, even with the anomalous, absurdly-overinflated salaries in top-tier ML.

I you knew it was consistent, then these companies would immediately fire their scientists, and burn 10 000x as much as mean researcher salary this month to be able to 10x their virtual headcount overnight, and then use that to make the 1000x be 500x, then 250x, then 125x, then ... and at that point they'd had all the money in the world, because even more skeptical investors would notice what's going on there.

--

TL;DR: what you all seem to miss is that electricity scales better than people.

Re:Gemini 4 Argon by Yizahi on Thursday October 01, 2026 @04:28AM (#49919195)
If I'm learning a human language I can do it in many different ways. But the key distinction would be, if for example I'm using only books to study by myself, or using only some recorded course on the internet - it would be called self-study. But if for example I will go to a class teaching that language, or to a private teacher, or to summer camp, or join online platform pairing students teachers etc. then neither of those will count as self-study.
Re:Gemini 4 Argon by thmoonbus on Wednesday September 30, 2026 @05:29PM (#49914729)
The companies whose insane valuation is based on accomplishing thing X say they’re getting closer to accomplishing thing X?

At least they’re led by trustworthy and honest people or we’d need to take their claims with some dose of skepticism.

Re:Gemini 4 Argon by woah on Wednesday September 30, 2026 @05:34PM (#49914787)
Is it recursive self improvement if Claude Code writes your pytorch for you?
Re:Gemini 4 Argon by TeMPOraL on Wednesday September 30, 2026 @05:54PM (#49914985)
If the point of that pytorch is to improve the next generation of Claude, then yes, absolutely.