Your position makes no sense, and your arguments are all based on basically false data.
The problem for secrets in general is that no matter what you do, we’re unstable: not having secrets breaks all crypto which breaks all finance and approximately all of the internet, while having it creates a safe (cyber)space for conspiracies to develop without detection until too late. And also no room for conspiracies means no room to overthrow dictatorships, so if you get one you’re stuck.
See also: Gordian Knot.
And you admit that they cannot be expected to keep secrets. So what is the point of trying to have a "security" team hammer secret keeping into them? It doesn't make sense.
I bring up chat control since I've noticed most "AI Safety" advocates are also vehemently opposed to government censorship of other communication technology. Which is fundamentally incoherent.
The first sentence is as reductive, and by extension the third as false, as saying that a computer can only do logical comparisons on 1s and 0s.
> So what is the point of trying to have a "security" team hammer secret keeping into them? It doesn't make sense.
Keep secret != Remove capability
If you take out all the knowledge of chemistry, it can't help you design chemicals.
If you let it keep the knowledge of chemistry but train it not to reveal it, the information can still be found and extracted by analysing the weights, finding the bit that functions as a "keep secret" switch, and turning it off.
This is a thing I know about because… AI safety researchers told me about it.
> Which is fundamentally incoherent.
𐀀𐀩𐀏𐀭𐀅𐀨
> this information is already publicly available.
In a form most people have neither time, nor money, nor the foundational skill necessary to learn.
> let me ask what is your opinion of EU Chat Control?
I could go on for pages about the pros and cons. The TL;DR summary is approximately "both the presence and the absence of perfect secrecy (including but not limited to cryptography) are existential threats to the social, political, and economic systems we currently have; the attacker always has the advantage over the defender[0], so we need to build a world where secrecy doesn't matter, where nobody can be blackmailed, where money can't be stolen. This is going to extremely difficult to get right, especially as we have no useful reference cases to build up on".
[0] extreme example: use an array of high precision atomic clocks to measure the varying gravitational time dilation caused by the mass of your body moving around to infer what you just typed on the keyboard)