It's a tool but it's not a strict computer program, and the language we use for computer programs doesn't capture their behavior well. In squishier fields like bio, where model organisms are used as living protein factories, there's a reason we don't call maladaptive behavior a bug. When dealing with useful but complicated black-box systems, we should feel free to reach for any metaphor we please.

Would it make you feel better if the user said 'It'll make significant changes to how the code works. In the COT, it brings up explanations that resemble anxiety about copying code, and my suspicion is that this is the proximal cause of the changes it's making.' -- this is inarguably true, and very likely what the guy you're responding to meant.

It very obviously is a strict computer program. It's a program executed by a computer, what else could it be?

It's a blob of data where only 1% of it was written with intent and the behavior of mimicking text has to be wrangled and coaxed far more than with a normal program.

That's the rough intent of putting the word "strict" into that sentence.

What is the difference to meat brains?

I mean you can try to threaten ssh with it's kids lives but I doubt it'll let you log in.

I mean besides the fact that it's, by construction, pseudorandom?

It's misalignment. Call that a bug if you like, but it's a different concept.

We can argue about https://en.wikipedia.org/wiki/Chinese_room all day... but when a thinking trace repeatedly says things like "Wait, should I avoid copying this directly?" despite a simple instruction, it's hard to describe that as anything other than "anxiety."

It has been shown that advanced llms have internal flags that behave more or less the same as emotions.

No way, this presupposes sentience in LLMs, because emotions are subjectively felt experiences.

It's completely different from encoding a relationship between an emotion and what kind of language it tends to coincide with.

It doesn't matter for this purpose whether it "actually feels" or not. It is modelling emotions because the training data models emotions because it comes from humans that have emotions, and that's a fact.

Seems perfectly cromulent to me. Do you think something is obscured or miscommunicated by this description? Of course it's a poetic way of writing, but it gets the point across.