News for hackers. Stuff that matters.
Re:Gemini 4 Argon by SwellJoe on Wednesday September 30, 2026 @05:29PM (#49914725)
Sure, it's happening...but, is IT happening? By that, I mean, we can see that the models are able to iterate at a pace and scale that humans can't match, and that provides gains in model performance and efficiency. But, humans are still needed in the loop, and not just because it's necessary for safety/alignment reasons. I don't think any significant discovery has been made by models on their own, and I don't know that LLMs will ever have the capacity to invent. They can synthesize from known data amazingly well, and since they know everything "known data" is extremely broad. But, the leaps, so far, have all come from humans.

So far, I don't think the models are capable of running away on their own. Of course, it would be playing with fire to not at least consider the risks of such a runaway scenario and build in safeguards against it. But, there is no model that can build a better model on its own, thus far, to the best of my knowledge (which is far more limited than the models, so maybe I should ask them).

Re:Gemini 4 Argon by TeMPOraL on Wednesday September 30, 2026 @06:00PM (#49915064)
Recursive Self-Improvement isn't instant, it starts slow and accelerates.

It starts with what they already claim to be doing - increasingly relying on existing models in non-trivial work related to training, evaluating and optimizing the next, more capable generation of models. As long as the proportion of work keeps shifting towards agents doing more and more of it, and humans less and less, that's RSI at play.

It may be that it turns out LLMs lack some fundamental level of judgement and it plateaus, but frankly I find this notion absurd; LLMs already show better judgement than most people. The alternative is, at some point LLMs will show the ability to futz their way into improvement of the next generation of models even without humans in the loop - even if much less efficient at first, if generation N+1 is more capable than generation N, it'll either take off or burn out.

Re:Gemini 4 Argon by freecodeio on Wednesday September 30, 2026 @06:48PM (#49915472)
you are really just talking out of your ass here, no offense

just because more and more agents are doing human work, that in no way means the model somehow becomes magically more intelligent, it just means the work will stall and continue on at the same level forever

hell even if they hypothetically have an internal model that can output the entire training data set in a better format, there's no scientific evidence that the newer format has new information that is sufficient enough to train a better AI

as a matter of fact the scientific evidence is on the contrary

Re:Gemini 4 Argon by TeMPOraL on Wednesday September 30, 2026 @07:13PM (#49915684)
It's not the number of agents you should pay attention to, but the scope and nature of work they can perform effectively without being micromanaged by humans.
Re:Gemini 4 Argon by gytt33 on Wednesday September 30, 2026 @07:44PM (#49915930)
He talks out of his ass on a lot of topics on here especially those concerning economics and finance.
Re:Gemini 4 Argon by iamarobot on Thursday October 01, 2026 @11:13AM (#49922763)
Even currently, there are clearly diminishing returns throughout the field. While not a skeptic myself, I think the age of LLM's is now over or close to plateauing. GPT-6.1 Sol seems almost the same as GPT-6 Sol(even if the benchmarks are slightly better). It doesn't help that we are running out of real training data, requiring new Synthetic data to be created for the next generation of Models. AI(or SI), has hit its peak, IMO, and now the focus shifts directly to Autonomous tasks. While Fable 5.1 is borderline okay at autonomous tasks, the amount of Slop it generates fundamentally requires a human to correct its mistakes. I was working on a website for a nonprofit with Fable, but the amount of slop and corrections I had to make was astounding/
Re:Gemini 4 Argon by breuleux on Wednesday September 30, 2026 @11:36PM (#49917435)
> It may be that it turns out LLMs lack some fundamental level of judgement and it plateaus

All intelligence, LLM or not, is bound to plateau around the point where the need to operate within physical reality bottlenecks the speed of feedback. AI is progressing swiftly in the digital realm where feedback is nearly instantaneous, but it's unclear whether that would translate into improvements in the physical world where signals are much noisier and intelligence and judgment are less impactful.

Re:Gemini 4 Argon by TeMPOraL on Thursday October 01, 2026 @02:59PM (#49925695)
Our tried-and-true approach for when the physical world is causing us trouble, is to remake the physical world to be easier to work with.

My go-to example: wheels suck at mobility in the natural world. There's a reason no animal uses wheels as their means of locomotion. They only really work on flat, hard surfaces that give a good grip.

Our solution? We didn't build mechanical legs for all-terrain mobility. We paved the world instead. Dirt roads, then brick roads, then asphalt and concrete - since ancient history, we were forcing the world to adapt, so the one means of mobility we could easily built was effective.

We repeat this pattern every time when the world doesn't agree with us, and adapting ourselves to it is too much of a hassle.

Now, AI will likely adopt the same approach to these kinds of problems, which may not end up very nicely for us.

(Note: we are already adapting our own digital worlds to make them easier for AI. Much like building roads for our wheels, we now have a resurgence of CLI, with new tools exposing high-level operations optimized for agents - not humans - to use comfortably.)

Re:Gemini 4 Argon by breuleux on Sunday October 04, 2026 @11:13AM (#49954597)
Sure, but is the limiting factor intelligence, here? Give human intelligence to dolphins and they're still not going to be able to pave the world. In order to remake the physical world, we needed certain physical abilities -- we needed the animal mind and body that took billions of years to evolve and embeds advanced priors about the world, and we needed it packaged properly. Irrespective of our intelligence, our body is a general world navigation device adapted to grab and move stuff. We use this innate tool to remake the physical world, but we didn't make it, and we'd be impotent without it.

With AI it's a bit of a chicken or the egg situation. Before it can ablate the complexity of the world, it needs to be able to deal with it, including adversaries such as ourselves. For this it needs a tool as sophisticated as our bodies, but could AI develop such a tool for itself without the help of this tool? I'm not convinced.