breuleux
The problem is that you have to simulate them accurately enough and fast enough. A perfect simulation of physics will necessarily be orders of magnitude slower than physics. To accelerate it, you need to figure out which parts can be cut or approximated without derailing the model, and just that part will require a lot of back and forth with reality.
Success isn't guaranteed either. If a physical system does whatever it does optimally enough, fast simulation is just physically impossible. Furthermore, even simple chaotic systems require unrealistic precision in the initial parameters to be useful. Natural entities are fairly chaotic and also pretty darn efficient at what they do, so it's plausible that even the best possible simulation of a middle manager is a slower bottleneck than a real middle manager.
There is also the problem of alignment, which is verifying that version N+1 behaves in a way that version N approves of. This is ostensibly the problem that we humans have with respect to AI development -- how to make sure they align with our values -- but it's also true of AI with respect to its own successors. Improving oneself without losing the plot would necessarily be slower than throwing stuff at the wall like evolution does.
"Wildly different?" That's a wild exaggeration, if you don't mind me saying. Subjectively, sure, there are lot of important differences. Objectively, they're basically the same. The intellectual environment machines bask in is many orders of magnitude more different from physical reality than living on Earth is from living on Jupiter. Even the topology of it is without common measure.
> and the reason why it was possible was because we have a tiny bit of general intelligence
Also, hands. Also, an upright posture. Do you think just any being with the level of intelligence we have would have followed the same path? Attributing our success solely to our intelligence is simplistic -- it is our intelligence plus a slew of innate capabilities, which we did not create, and without which our intelligence would be worthless.
> For your simulation argument, see e.g. https://alicorn.elcenia.com/stories/starwink.shtml
The kinds of simulations you find in LessWrong's half-baked thought experiments are almost always ridiculous and this is no exception. "World simulations" in real life are massively limited stuff like Minecraft where you might be able to embed a 1 MHz computer if you manage to bump it to run 20K ticks per second. The resources required for the simulated beings to do all that work wouldn't be available and their generations will be slower than ours, not faster.
I didn't spell it out, but the simulation I had in mind was for a toy world, because only a toy world could possibly run fast enough.
> I think you're arguing that nothing like this could possibly happen, because humans are too dumb to ever figure out unfamiliar physics without prolonged experimentation
I'm talking about operational, practical, instinctual knowledge for real-time action. Knowing physics doesn't make you capable of catching a ball, and you don't need to know physics to catch a ball. In real life, we never deal with the fundamentals of physics, we deal with macroscopic objects and systems that are very far removed from these fundamentals. They are more varied, messier, faster, and overall harder to deal with.
> I think there's a fundamental difference, between the sort of careful planning that lets you launch space probes, and blind optimization like evolution does. The former can generalize to unfamiliar domains even when the latter doesn't. That's what intelligence gives us.
Blind optimization has one definite advantage over careful planning, however: it is not self-limiting. Any goal-directed optimization process is de facto limited to the subset of possibilities that it can prove furthers its goals. That's why most human technology is simple and unsophisticated compared to natural systems: it needs to be predictable, reliable, build-upon-able, and therefore easy to model.
And when it is more sophisticated, like current AI, it is so heavily inspired from natural optimization processes that I would argue it is actually a form of blind optimization. I mean, think about it: we understand AI so poorly that you and a lot of other people are afraid of losing control of it. If we were designing it intelligently, we wouldn't be worried. But we aren't: we are doing it blind.
I think the same is true of "research" and intelligence in general. There is no such thing as a general intelligence or a general capacity to research. It's always fine-tuned to an ambient setting, including ours. Even within humanity, our intelligence is most effective in fields where rules are rigid and quick and decisive feedback is available (math, engineering, computing), but it is much less effective in soft sciences, and disappointingly useless in politics. In other words, the dynamics of innovation and improvement are not purely mental, they are very much tied to external factors that are not always easily controlled.
> it'd also require "physical intuition" to be a harder to generalize skill than research, which also seems like it can't possibly be true - one of these is just physics
It's not "just physics", it's a honed system for surviving a realistic range of physical conditions, assess material properties on touch, adapt manipulation to arbitrary shapes, predict trajectories, take into account variations in ground, smell, wind, and so on; not to mention the machinery of the body itself, which self-repairs and protects us against a range of complex threats.
The point isn't that machines couldn't do all this. The point is rather that you can't quickly iterate and adapt designs to do it from an ivory tower. Pure "research" simply won't get you there, you also need to experiment, and insofar that physics limit the throughput of these experiments, the speed of improvement will be throttled severely. It is a mistake to think that "RSI" could break this, because it is categorically the wrong lever to pull.
As impressive as AI might be at virtual tasks like coding, virtual reality is remarkably different from physical reality in ways that artificially inflates AI results. One, it is far simpler. Two, the feedback loop is far, far quicker. It's possible AI could get the animal intelligence and animal body that lets us humans actually apply our intelligence in reality, but nothing in current technologies really indicates that this is a given.
AI doesn't need to be smart to seize our resources. They need brute strength, a proper physical intuition, and "hands". That may very well be several orders of magnitude harder than anything they're currently doing, we're just lucky it was in our starting build.
Even if that's true, quite a bit rides on how long "longer" is. If "longer" is a million years, that's not exactly good news.
As far as AI goes, I think that whatever it is that they have is analogous in significant ways, but also irreconcilably different in significant ways. They aren't proceeding in a straight line towards consciousness either, because why would they? They're doing their own thing which happens to produce similar outputs. It's a problem in terms of value: we value consciousness, because we value ourselves, because we value our output. If it turns out consciousness as we know it is just a minor implementation detail with no cosmic or long term significance, then irrespective of whether AI is conscious or not, we still have to grasp with the challenge of explaining why consciousness is valuable in the first place if not for its outputs.
I'd argue that the reason why we value life, consciousness, the ability to suffer, was always arbitrary, and it relied a lot on the unicity of human experience. It was never robust to the breaking down of human capabilities into inspectable, reusable, upgradable and resettable modules. Or the radical decoupling of capabilities we hitherto considered correlated, like writing about love and feeling love, and many other things.
I disagree it's unoriginal, though. It's pretty original, you just have to be prepared to suspend your disbelief (it's kind of on me that I wasn't when I read it).
The chief issue with external incentives like "fame" or "money" is that they will attract a lot of people who care about the reward, but not the work. If they can find a way to cut corners, they will. So it works fine if you have a very tight, objective and reproducible evaluation criterion such that you can't get rewarded for poor or sloppy work, but even in science this is difficult to establish, especially in a crowded field where most advancements will be small and incremental. Science works a lot on trust, if only trust that the described experiments were indeed made and the results were indeed as reported.
That's why I feel we get the best science when it's driven by people who just intrinsically want the best science, because getting it right is a reward in and of itself. It'd be neat if it was possible to identify them and just let them cook, regardless of whether results are produced or not. Paradoxically, I think this may be easiest in a setting where academia is undervalued -- the knowledge-seekers will always self-select into research (provided the conditions are good enough and they are not too undervalued), but the fame-chasers will have to find their validation elsewhere.
With AI it's a bit of a chicken or the egg situation. Before it can ablate the complexity of the world, it needs to be able to deal with it, including adversaries such as ourselves. For this it needs a tool as sophisticated as our bodies, but could AI develop such a tool for itself without the help of this tool? I'm not convinced.
Do they? Business valuations are more a reflection of what people believe will yield returns than a reflection of reality. A lot of big tech behemoths could disappear from the face of the Earth overnight and it would make little difference to our lives.
Natural resources and energy are the actual backbone and it is a true catastrophe when these are disrupted. In comparison, disruption to compute or data is just an inconvenience.
All intelligence, LLM or not, is bound to plateau around the point where the need to operate within physical reality bottlenecks the speed of feedback. AI is progressing swiftly in the digital realm where feedback is nearly instantaneous, but it's unclear whether that would translate into improvements in the physical world where signals are much noisier and intelligence and judgment are less impactful.
For what it's worth, ten thousand terawatt fusion plants probably approaches the level at which the sheer intensity of energy production would cause significant disruption to the climate (it's roughly 5% of the Earth's entire solar input). Every energy source becomes dirty past a certain point. It would be wiser to learn how to build a utopia within a limited energy budget than find a way to produce enough of it to cook the damn planet, but who am I kidding, we're going to build a million of these things.
Intelligence isn't magic. It can only overcome so much brute force and humanity, as a non-intelligent system made out of intelligent parts, carries an ungodly amount of brute force.
If that's the case, unfortunately, they're bound to find legitimate fraud eventually. Given the subsequent hard evidence of plagiarism, it seems like this is one such case, and we can't exactly ignore it.
Comparing to other Haitian authors was also relevant, because another defence from the author was that his "caribbean style" could have tripped up the detector.
But yes, other than that, that's a good point. My anecdotal evidence is that my own unpublished writing, both in French and English, contains no false positives, but even in that case, that's only evidence that my style isn't AI-adjacent. I'd be curious to see concrete examples of false positives, but it's hard to fully trust these if they don't have proven timestamps.
* Media ran the entire novel through Pangram and got a 94% AI score [1], whereas all other Goncourt finalists, Haitian/African novels and French classics they tested came out as 100% human. I recall they also tested works specifically from before and after the AI boom and found a shift, but I don't remember where I saw that.
* People on social media, as well as traditional media, have then found cases of blatant plagiarism in Orelien's work a decade ago (this includes one prized short story and several posts to blogs and media) [2]
I think we have to be careful not to assume that tools like Pangram are completely accurate and the word of God, but we can't dismiss them outright, and this particular author's history of plagiarism makes it much harder to give him the benefit of doubt.
[1] (French) https://ici.radio-canada.ca/rci/fr/nouvelle/2286476/thelyson...
[2] (French) https://www.lapresse.ca/arts/litterature/2026-09-23/thelyson...
Wiping out an occupation that people enjoy and that gives them purpose (making them "obsolete") is a public mental health crisis, if anything. It's unclear that the "benefits" of the "solved" field actually outweigh that.