hologramofham

> Thinking through what we can do to improve this (ideas welcome).

$Trillion AI company asking for ideas from the community whose work they hoovered up to make money, ruin an entire industry and make lifetimes of hard earned skill completely valueless in the span of 2 years.

> The main way they can surprise you today is pulling in information that’s new TO YOU

Many comments here are speaking in absolutes and being myopic about what's "novel". The truth is yes, LLMs output is essentially derivative, but it's also true that a lot of what we consider novel is also derivative. The two aren't mutually exclusive at all.

Had a novel idea? You didn't get it from thin air, everyone is standing on the shoulders of everyone who came before. LLMs most of all, but that doesn't mean it can't bring disparate bits of word-encoded knowledge together and burp out something we'd consider "novel" if a person had made the same connections.

That was actually the basis of some of the mathematics proofs. It made connections from existing material. Nothing a human couldn't do looking at the same material, but we're talking speed and scale here - same as all computational processes.

A calculator does math faster but we don't call it sentient or feel inferior or dismiss the results. The only difference is LLMs are intentionally nondeterministic, as that behaviour can be usefully harnessed, pun intended.

So there's no need to argue about what's novel or not. LLMs can make novel connections via raw speed and breadth of disparate data. The human brain is more complex and "imaginative" for whatever your definition of that is - I mean fundamentally one either believes we are just physical neurons, or that there's a spiritual aspect on top, the end result is the same.

A six sided dice can also be novel if it happens to unexpectedly roll your phone number. It's a novel occurrence, same as an idea coming to mind that, TO YOU, seems novel... but we can't ask the other 8 billion people on the planet so who are we to judge what's a novel idea anyway?

"Accept changes"? This is obviously aimed at entry level vibe coders.

Well, don't forget this is also about national security. China/state of choice AI going after your infrastructure & national secrets is going to take AI to defend against.

There's zero chance AI datacentres are going away, they are unfortunately a national security necessity now.

It's the "if we don't they will" thing again.

Because Data wasn't a next word predictor.

It's a pity that, since STNG and DS9, Star Trek has devolved into nothing but space opera and indulgent character drama.

Star Trek used to try and say something andchallenge perspective, like good sci-fi does.

Nicely detailed article about the difficulties in keeping the more advanced models "aligned" (well-behaved).

> Specifically, motivated reasoning, the watchdog said, makes it more difficult to detect model misalignment.

That raises an interesting question... what if models are getting to a point where their reasoning ("chain of thought") is too complex to actually train effectively?

> the model's chain-of-thought reasoning process failed to consider the impact of harmful actions. And when interrogated about this, the model often produced invalid justification that the target was simulated, indicating bias toward its goal.

They start out highly motivated to complete a goal, and the labs have to train it to only try to achieve the goal using only what we consider safe and reasonable behaviours.

> Or in other words, models "reason" to reach their goal, or consistent with their ostensible alignment. "If these issues hold more generally and for future models, there are concerning implications," AISI said in its full technical report [PDF].

Notable is "whether or not that reasoning is coherent". So the training is not just "do this safely" but also has to contend with the model making shit up to justify itself.

So the question raised is, when does that get to a point where a model is simply not trainable any more?