mikestylz

> Should we assume that all software is more compromised than the kernel?

Absolutely, the front door has always been unlocked. We were previously protected by the economics of exploit detection favoring the developers, but now we have entered a new world.

The issues arise almost everywhere we look. It's frustrating because the user experience in edge systems has barely changed in the last fifteen years. We could have rode the advances in semiconductor manufacturing to embed more layers of protection such as CHERI while slightly slowing down the rate of performance improvements. Instead we have only marginally upgraded security and largely added features which the users scarcely notice.

I am hopeful about the long term prospects of formal verification, but having humans reviewing the proof statements and confirming they match our desired properties will take many years. Meanwhile, AI assisted hacking is an urgent issue. So we will have to delegate implementation, proof specification, and proof development to the AIs in the short term. Maybe that goes well, but who knows. Interesting times.

Athletes continue to break records despite manual labour being unnecessary for most jobs. And fitness culture has never been more prevalent. We're competitive creatures and this is an ability we are hardwired to appreciate. I see no reason why mental fitness won't see the same development in the future.

A few years ago I was in a pure math degree, but I switched and graduated with a combined degree in math and computer science. Once I finish my masters degree I intend to work on industry applications of formal methods, which I anticipate will be a high-growth area for the next few years. The idea is to take advantage of the growing ability to generate proofs/code and work on keeping the messy human side of the equation up to speed. So I plan on addressing the social and operational aspects which come with delegating more work to AI. I'll work on integrating these gains into the industry, on building trust in the work that AI produces, and I'll work on the interface that lets us ensure that this trust is deserved.

But nothing is certain. I am trying to avoid overspecialization and remain flexible, just as you advise. If this path fails I might become a paramedic or physiotherapist.

Thanks gramps, you're right I should have gone with the PID controller analogy.

Just about anyone who has played a sport has heard that they have to target where the ball is moving to, not where the ball is right now. In this regard I am consistently disappointed with the commentary that mathematicians have been producing lately.

As a student, I am actively making decisions which will shape my career for the next forty or so years. At a minimum a discussion like this should acknowledge the possibility that the current rate of AI progress continues apace. I understand the desire to be encouraging, but the best preparation for students involves the consideration of possibilities that current mathematicians negligently paper over.

Of course it will get better at exposition than humans. And it will prompt itself in due time.

We can't stop it. But you can use your voice to buy time and resources for alignment and safety research. A few additional months may make a world of difference.

> When AI becomes more capable of navigating practical problems without human intervention, I will start to be concerned.

Given the past rate of progress, why not start being concerned now? It's a bit like the economist saying that the optimal number of flights to miss is not zero. If you keep landing short on your estimations for how far this technology will go, next time you should err on the other side.

And regarding Vending-Bench 2 (https://andonlabs.com/evals/vending-bench-2) my understanding is that models do pretty well on it now.

In the past we might have said horses, stone, and bronze were three things necessary to advance civilization. In the future I'm sure the AIs will prompt themselves with nice problems. But for now I agree with AGMAI that we should be careful not to disrupt the field too severely. It's reasonable to keep it around even if it's just functioning as a temporary "strategic reserve of expertise"

You seem like a smart person. I'm glad you're pushing back but cool the tone, and let's see if we can get to an agreement. There is not much evidence for these trade-offs put forward in the article, just on the order of five instances in nature which form a kind of curve if you squint. Should this be convincing? Well there are even definite outliers in history. Take smallpox on the native americans, or the bubonic plague. Ten years ago a mysterious virus wiped out 90% of the world's Bellinger River turtles. Why aren't these points in the article? Before the invention of the airplane there were many biologists saying such a thing was impossible, because they were extrapolating from the example of avian flight.

As for complete satisfaction with arguments, I'll admit you're right. I am often reasonably convinced but I don't know if I would ever give 100% confidence to anything. There can still be loads of zeros or nines after the decimal though. What's concerning here is that the article speaks in absolutes. I'm inclined to thinking in expected values. Even if we believed there were only a 1% chance such a virus could ever be designed by an AI, if the effect were to be a global plague of biblical proportion, we should still take that possibility extremely seriously. There's no humility shown in the writing, and that's the single most problematic part to me.

It is not, because we should not expect a manufactured technology to behave exactly as an evolved one. My understanding is there is not much incentive for a virus to kill off all its hosts. When they do it is largely an accident, and it is not uncommon for a severe virus to eventually get replaced by a less virulent and thus more successful strain. It's pretty hard to hill-climb evolution to such a point since the virus could starve itself in the process. And induction on evolution feels misguided to me. Consider our species. A million years ago it would have seemed completely reasonable to state that the absence of advanced tool use in the kingdom of life suggests it is biologically impossible. After all, such a thing had not occurred in hundreds of millions of years!

To be clear, I am not saying that such a virus will exist eventually. I am just noting that the arguments displayed here are not completely tight, and there are unknown unknowns at play. Some things are unfalsifiable. This is common.

The magical level of intelligence required is not really a flaw of the argument's deductions, it is its very premise. If you believe that there is a possibility of an AI working on itself, you get the approximate differential equation d/dt f(t) >= C*f(t). In plain English, at least some proportion of development's progress goes back into future development. The (least case growth, assuming equality) solution to this is an exponential function, and exponential blowup leads to stupendous and magical seeming results. I then ask - what is evolution if not a dumb optimization process? How are we supposed to believe that it devised miracles such as the human mind, insect flight, echolocation and so on while denying the possibility of a future EXTREMELY power simulation to do the same? I went in really wanting to be convinced by this article for the sake of my own mental health, but I am disappointed. What we really need for such a statement is concrete physical evidence that a virus is simply incapable of having such facilities. Or perhaps an equivalence established showing biological defense is no harder than offense. Until it exists, I will continue to worry.