LinchZhang

HF was OpenAI's agents not Claude.

As I said multiple times in the linked post, everybody knows that CoT monitoring is imperfect.

The question is whether the benefits are high enough that you should be happy to drop an imperfect safety mechanism in the hopes that some day, one day, you'd get a better one. Seems like the wrong tradeoff to me but regardless, arguing that "CoT monitoring is imperfect therefore we should drop it" when we do not in fact have a better mechanism in place is silly.

This is not how we do things in any other engineering discipline or risk-mitigation system.

It's like saying we shouldn't have rapid antigen tests because they have nonzero false negative rates, or not writing software tests because the tests never catch all bugs.

> Not everyone in the AI space approves of these people or their doomerish.

"Not everyone in the automobiles space approve of consumer safety advocates" What an understatement!

It's very hard in practice. Even without optimization against it, it's about as hard as understanding an activation layer today and will probably get harder in the future.

There's also nothing stopping the secondary LLM from confabulating bullshit and there are less checks on it than CoT (harder for either humans or other models to externally verify).

I agree in the long run whitebox interpretability would be better than CoT monitoring but the technology is very much not ready for it today (and it's not clear we'd solve enough of interpretability before the AIs have actually scary capabilities).

I agree CoT monitoring is imperfect but it's okay in practice and helps us get defense-in-depth re: model intent. We absolutely do not have a singular safety mechanism in place that's sufficiently good that we can use it in exclusion of all other imperfect ones.

"Do you know what is a faithful representation of what the model wants to do? Tool calls."

tool calls could absolutely be spoofed, my impression is that this happened many times in the OAI HuggingFace attack.

"Phoebus keeps taking screenshots of our latest model’s thoughts. It’s getting kind of embarrassing."

Thank you for your kind words and empathy! I appreciate it. Writing to the void is hard, and while I care a lot about improving and not being wrong, I also appreciate it when people realize that the writer on the other side of the screen is a real person, and deliver their feedback with kindness and empathy.

Thanks, appreciate the anecdote!

If you have suggestions on a better title, please let me know! I tried pretty hard to come up with different ones, giving time constraints, and this was the best one I had. I'm really bad at titles and this is an area of active growth for me :)

Why fixate on a specific word rather than the overall idea? If the idea is clear enough I don't think it's worth fighting over semantics.

I agree I used the word substantially more expansively than some other people use it. That's why I defined it in the beginning so people can understand the local scoping of the relevant word! :)

(That said "salespeople" are in the middle layer under your definition as well)

The other term I was thinking of using for this post was "bullshit jobs." So titling my post "bullshit jobs are real jobs" but I didn't want to fight against the motte-and-bailey of specific jobs being possibly bullshit jobs.

("coordinators" presumed the conclusion too much and also points to a specific thing )

I think many people have some intuition that work can be separated between “real work“ (farming, say, or building trains) and “middlemen” (e.g. accounting, salespeople, lawyers, bureaucrats, DEI strategists). “Bullshit jobs” by David Graeber is a more intellectualized framing of the same intuition. Many people believe that middlemen are entirely useless, and we can get rid of (almost) all middleman jobs, RETVRN to people doing real work, and society would be much better off.

Like many populist intuitions, this intuition is completely backwards. Middlemen are extremely important! Coordination problems are real problems, and the bottlenecks to global wealth and flourishing.

The post goes into details for why.

The supposed excuse was Elon Musk's lawsuit, but they interpreted the discovery process very broadly.

I compiled a list of my favorite intellectual jokes, as well as offer a short treatise on why intellectual jokes aren't just "jokes about smart people"

In xenosociology class we learned about a planet full of people who believe in anti-induction: if the sun has risen every day in the past, then they think it’s very unlikely that it’d rise again.

As a result, these people are all starving and living in poverty. An Earth xenosociologist visits the planet and studies them assiduously for 6 months. At the end of her stay, she asked to be brought to their greatest scientists and philosophers, and poses the question: “Hey, why are you still using this anti-induction philosophy? You’re living in horrible poverty!” The lead philosopher of science looks at her in pity as if she’s a child, and replies:

“Well, it never worked before…” __ Did you know? The moon landing was staged. It was faked by Stanley Kubrick.

But Kubrick was a perfectionist, so he insisted that they shoot on location.

I have heard approximately equal number of comments that Chiang's writings are clearly hard sci-fi and that his stories are clearly science fantasy.

I find my own ontology to be useful for myself and others, but I don't begrudge others their preferred definitions.

If you like explorations of the deep impacts of parallel universes which have bridges that lets you transport bits (but not matter), you might enjoy https://brainchip.thecomicseries.com/

I like some of Yudkowsky's shorter fiction but I could never get into HPMOR. The writing style just comes across as too smarmy. I know it's intentional but I still can't get past it (I had trouble with Thomas Covenant) as well.

For this article, I wrote 4000+ words in the first draft and asked AIs to help suggest places where I repeated myself, different sections or other things to cut, etc, so the final article is tightly focused. I tend naturally to be longwinded, and it's helpful to go through iterations of figuring out what to cut in consultation with AI. I also use them to check grammar, typos, and spelling, and whether my points are too complicated (as a rule of thumb, if I didn't explain something well enough for Opus to understand, I assume I need a better explanation for human readers as well). I'm not theoretically opposed to having AIs do the writing but empirically I have not found them useful.

This might be presumptuous of me, but I do not believe current-generation AIs are capable of writing articles of the quality level of my nonfiction writing. I think they have a) lower quality and quantity of overall insight, particular on philosophy-adjacent topics, b) lower ability to be appropriately confident in making claims well[1], and c) noticeably worse ability to "write well", subjectively defined (eg weaker sentences, less deft use of metaphors and references, tries too hard to force a point when there's nothing there, etc).

Honestly I find myself slightly offended by the comparison, though I acknowledge it's one of the things where 2-3 generations down the line AIs might well surpass me at.

I think "blurry jpeg" misses the point, for most practical questions. It conflates substrate and developmental history with emergent capabilities and consequences, and is only one step up from saying "AI is just 1s and 0s"

[1] To be clear this is something I have not mastered, I'm just saying AIs never seem to get this well, though maybe I'm just bad at prompting. In particular there's a particular "internet slop"-style that they go to very quickly, and when I try to prime them away from that they sound fake in a different direction.

I'm so glad my review had an effect on people! I hope the book can bring you half as much joy as it brought me <3 <3 <3

checking logprobs doesn't seem weird to me, it was the priming that was weird.

It wasn't one of the short stories I reread for the review. I thought he simplified the thermodynamics element to make the story work, but multiple people have corrected me by now. Note the specific wording was "appear to" because I wasn't sure.

This seems like a weird way to check if something's AI? a) Like presumably AIs are much more likely to make mistakes of a certain form if there are more such mistakes in the training data (or similar ones) b) to figure out whether something's written by AI you want to figure out if AI can independently generate it rather than heavily be tricked to make a specific mistake.

It's really cool that you ask 10 people their favorite chiang story, and chances are, you'd get 11 answers. And he didn't even write that many more than 10 stories!

Really tells you both how talented he is, and how different stories just speak to different people.

Well, feel free to send my review to anybody cool living in SF or East Bay, especially people new to the area! Maybe they'd read the review and think they'd vibe well with me :)

OMG I'm so glad this review might have an impact! Please do check out Story of Your Life and then read the other stories!

Without giving too many spoilers away, the short story's plot is simultaneously extremely similar to and extremely different from the movie. YMMV on which one you prefer, fans are divided.

In my experience people who read the short story first prefer the story, and people who watch the movie first prefer the movie. But you might be different! Just read it first and report back what you feel!

I appreciate the nitpicks!

Re #1 It's been several years since I read up on that area of philosophy. I'll need to reread some stuff to decide whether I think the definition I used is a fine enough simplification for sci-fi readers (and, well, myself) vs whether it missed enough nuances that it's essentially misleading.

(Some academic philosophers follow me on substack so maybe they'll also end up correcting me at some point!)

Re #2 ah I don't think of it as "sneaking in". It's more like "this is a view I have, this is a view many of my readers likely also have, given that this is a widely debated topic (as you say) and I'm not going to change anybody's minds on the object level I'm just going to mention it and move on."