jagraff

With driving specifically - human-driven cars are perhaps the greatest danger to my life. I very much would prefer to live in a future where human beings are not in charge of piloting multi-ton vehicles at life threatening speeds in the same areas I walk around.

By the way, all hardware has bugs too; we just don't see them often because the programmers building the stacks that we run our code on work around them.

That's a beefy MBP! I imagine it was quite hot by the end of that flight if you were running Qwen the whole time

As far as I know the circular deal fears are more around the chip purchasing agreements, because there is a speculative component to building out datacenters for training which may not produce models that are worth the training cost, but the revenue from inference is real and it has truly grown a lot over the past two years.

I personally believe that exercise is overwhelmingly beneficial and exercise every day, but there is evidence that athletes have slightly increased risk of ALS (though this may be related to something other than exercise itself, eg maybe the fields on which sports teams play have harmful chemicals, higher head injury rates, etc)

What I'm saying is that that is not sufficient to stop future harm; I expect the total damages would be less than the cost of a full training run, so it would effectively just be the cost of doing business. Liability is not sufficient to protect the world from dangerous technology - we need proactive rules around how the technology is developed, tested, monitored, and deployed, as we do with other dangerous industries such as airplanes, nuclear reactors, weapons manufacturers, etc

In other words, the risk of harm doesn't need to be zero, just less than the equivalent risk of a human with similar skillset. So I'm comfortable riding in a waymo, and not comfortable giving chatgpt my SSN at this moment in time, but I expect that within 5-10 years (assuming no doom) I will trust some AI agent with my SSN because they will be better at handling sensitive info than humans

I’ll admit I also just don’t understand the idea that someone saying their product is dangerous and could kill all of humanity is doing marketing - I just really don’t understand at all how that could be a marketing strategy. So I tend to believe they are saying it is dangerous because they think it’s true. But maybe I’m just naive.

Also I'm not describing what I personally think is an acceptable job; just that I understand that sometimes people do jobs that they think are wrong because they need money

Gradient descent/backpropogation is similar to evolution, in that both are optimization processes that over time discover better more efficient solutions to problems. The difference is that evolution is blind, and can only make progress via random mutation and natural and sexual selection, whereas backpropagation allows much more rapid discovery because it is directed

What I dispute is that AI agents are simple tools. I think rogue is an accurate word to describe them; I think what OpenAI is doing is more akin to gain-of-function research on a dangerous lifeform. I think this attack would have been prevented by air-gapping, but that wouldn’t solve the fundamental issue which is that they are creating something dangerous that they have no idea how to control

In the AI case, no, because I think the engineers believe that there is a high probability of enormous upside as well, if it doesn’t kill us all.

I did not imagine that the level of sophistication shown in this attack would be possible so soon; nor did I expect that agents would have goals so strong that they would attack a third party in order to achieve those goals.

I do know that some people predicted that cyberattacks like this one would happen; it seems like most of those people believe that AI agents do truly have internal goals, misaligned with their creators goals, and that they may end humanity after they exceed human intelligence and begin to self improve at an accelerating rate.