Turn_Trout

> Some experts sure. "Some experts" say lots of things.

I get this is Hacker News so we have to hate AI safety, but you should read more on this topic rather than blithely casting the x-risk viewpoint as fringe. The last hard data we have on the question is a late-2024 AI Impacts survey [1] of 1,580 authors at top AI conferences, who were offered $50 for answering a survey on the topic of the future of AI progress. The median respondent assigned a 5–10% chance (!) to "extremely bad (e.g. human extinction)" outcomes.

> On the factual end of things, AI safety experts have been raising the alarms for over a decade and so far essentially none of their predictions have come true.

Devastating. When I wrote my AI alignment PhD thesis [2] in 2022, and predicted a 70% chance existential risk by 2072 (which I later came down to about 1/3 on), you're right that that prediction hasn't come true yet.

While I agree many specific worries didn't materialize (like AI being like a "literal genie" it's hard to specify goals to), many key worries have (Hugging Face rogue AI swarm scans as cribbed from rationalist AI scenario writing).

[1] https://aiimpacts.org/wp-content/uploads/2026/09/ESPAI2024.p...

[2] https://arxiv.org/abs/2206.11831, top of page 4

> None of those amount to a ban

Please check your facts. Hegseth originally stated [1] that it would apply to ALL usage by companies that work with the military.

[1] https://www.cbsnews.com/news/hegseth-declares-anthropic-supp...

Yeah it speaks poorly of HN that they upvoted this confident nonsense.

> Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the first of the Anthropic employees going viral in the media) via a scholarship.

This conspiracy theory is truly crazy. A $20K scholarship in 2022 is supposed to explain Coxon walking away from unvested equity for a company worth over $950 billion dollars?

I'm one of the whistleblowers (from GDM [1]). I gave up over a million dollars (compared to quietly switching labs and continuing to work at one) to speak frankly about these issues. I hold no equity and tried to zero out my position before ever joining GDM.[2]

It's wild to me that people think such whistleblowers are fronts for labs to take over or pump valuations. We are trying to call out how these labs will, by uninterrupted AI-race default, concentrate enormous power over the rest of humanity.

[1] https://turntrout.com/why-i-left-google-deepmind

[2] https://turntrout.com/deepmind-equity-discussion

Dario's post [1] commits to direct evaluators that can, among other abilities, expose secret RSI. He wants that made law.

Do you have a source on METR employees retaining massive equity stakes?

[1] https://darioamodei.com/post/we-must-pace-the-frontier

OAI could check whether those accounts enabled training data. If "yes", OAI could trace whether that data was used in any related training process. If either of those answers comes out to be "no", then that's sufficient to conclude training data independence.

We wouldn't need a full ablated re-training and solution attempt, contra tedsanders in a sibling comment.

See also: https://turntrout.com/self-fulfilling-misalignment (my post)

As an aside, does the Waluigi Effect actually exist? My impression is it doesn't.

Those topics aren't on-topic for the essay. I've taken a pledge to donate at least 10% of my money to charity / impactful giving. A good sum of my donations have targeted high-impact opportunities to improve life for people in third-world countries.

Thank you for your praise.

> I'd guess TurnTrout doesn't agree on that framing, otherwise he probably would not have been at Deep Mind. But clearly he and I agree on other ethical positions; I am nothing but glad to see him stick to his principles here.

FWIW I agree that creators should be compensated (but evidently it wasn't a deal-breaker for me joining). I think it's bad how little that has happened.

I joined GDM to work on AGI safety to reduce existential risk from AI. I consciously avoided work that would improve the raw capabilities of Gemini. When I joined, I made a trade-off and it's valid to disagree with my decision.

I agree that they called many things remarkably well! That doesn't change the fact that AI 2027 is not a thing which happened, so it isn't valid to point out "this killed us in AI 2027." There are many reasons to want to preserve CoT monitorability. Instead of AI 2027, I'd point to https://arxiv.org/html/2507.11473.

AI 2027 is not a real thing which happened. At best, it is informed speculation.

No one has empirically validated the so-called "most forbidden" descriptor. It's a theoretical worry which may or may not be correct. We should run experiments to find out.

As someone who did their PhD in RL and alignment, it was not obvious to me a priori if, or when, or how badly obfuscation would be a problem. Yes, it's been predicted (and was predicted significantly before that Zvi post). But many other alignment fears have been _predicted_, and those didn't actually happen.

I don't think the existence of specification gaming in unrelated settings was strong evidence that obfuscation would occur in modern CoT supervision. Speculatively, I think CoT obfuscation happens due to the internal structure of LLMs and it being inductively "easier" to reweight model circuits to not admit wrongthink, rather than to rewire circuits to solve problems in entirely different ways.

> The #1 comment says that the rationality community is about "trying to reason about things from first principle", when if fact it is the opposite.

Oh? Eliezer Yudkowsky (the most prominent Rationalist) bragged about how he was able to figure out AI was dangerous (the most stark Rationalist claim) from "the null string as input."[1]

[1] https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...

They ran (at least) two control conditions. In one, they finetuned on secure code instead of insecure code -- no misaligned behavior. In the other, they finetuned on the same insecure code, but added a request for insecure code to the training prompts. Also no misaligned behavior.

So it isn't catastrophic forgetting due to training on 6K examples.

I'm the author of the GPT-2 work. This is a nice post, thanks for making it more available. :)

Li et al[1] and I independently derived this technique last spring, and also someone else independently derived it last fall. Something is in the air.

Regarding your footnote 2 re capabilities: I considered these kinds of uses before releasing the technique. Ultimately, practically successful real-world alignment techniques will let you do new things (which is generally good IMO). The technique so far seems to be delivering the new things I was hoping for.

[1] https://openreview.net/forum?id=aLLuYpn83y

First author here. Thanks for your comment!

> there's a lot hidden in the "if physically possible" part of the quote from the paper: "Average-optimal agents would generally stop us from deactivating them, if physically possible".

Let me check that I'm understanding correctly. Your main objection is that even optimal agents wouldn't be able to find plans which screw us over, as long as they don't start off with much power. Is that roughly correct?

> Theories on optimal policies have no bearing if

See my followup work [1] extending this to learned policies and suboptimal decision-making procedures. Optimality is not a necessary criterion, just a sufficient one.

> if as we start understanding ML models better, we can do things like hardware-block policies that lead to certain predicted outcome sequences (blocking an off switch, harming a human, etc.)

I'm a big fan of interpretability research. I don't think we'll scale it far enough for it to give us this capability, and even if it did, I think there are some very, very alignment-theoretically difficult problems with robustly blocking certain bad outcomes.

My other line of PhD work has been on negative side effect avoidance. [2] In my opinion, it's hard and probably doesn't admit a good enough solution for us to say "and now we've blocked the bad thing!" and be confident we succeeded.

[1] https://www.alignmentforum.org/posts/nZY8Np759HYFawdjH/satis...

[2] https://avoiding-side-effects.github.io/

Maybe you should read the paper, and/or the reviewer threads (as we discussed the nomenclature, and eventually agreed that "power" was accurate). We straightforwardly formalize a mainstream definition of power and show how it's more intuitive than the current standard measurement (information-theoretic empowerment).