If the U.S. doesn't control the weights, though, it can't restrict China from accessing the models...
1: https://thefuturemedia.eu/new-u-s-rules-aim-to-govern-ais-gl...
China didn't yet made a sovereign move on AI, besides investing in research/hardware.
There is no CCP master plan for open models, any more than there is a Western master plan for ignoring Chinese models only available as an API.
(not that I disagree)
The difference between the CCP, where "private" companies must actively pursue the party's strategic interests or cease to exist (and their executives/employees can be killed), and the US, where neither of those things happen and the worst penalty for a company not following the government's direction (while continuing to follow the law, which should be an obvious caveat) is the occasional fine for not complying with regulation or losing preference for government contracts, is categorical.
Only those who are either totally ignorant or seeking to spread propaganda would even compare the two.
https://triviumchina.com/research/the-ai-plus-initiative-chi...
1: https://www.theguardian.com/world/2023/jan/13/china-to-take-...
2: https://www.globalneighbours.org/chinas-zhipu-ai-secures-140...
It's also true that Anthropic and OpenAI have post-training that censors politically charged topics relevant to the United States. I'm just surprised you'd deny DeepSeek does the same for China when it's quite obvious that they do.
What data you include, or leave out, biases the model; and there's obviously also synthetic data injected into training to influence it on purpose. Everyone does it: DeepSeek is neither a saint nor a sinner.
Just because everyone does it doesn’t mean one isn’t a sinner for doing it.
I'd assume that when the Chinese do bad things people in China feel the same way about that as folks in the US feel about the US doing evil stuff, which is to say "very little at all". Why would they need to lie, any more than the US needs to lie? Do the average Chinese folks have more conscience then the average US citizen?
What a nonsensical thing to say. The CCP ruthlessly sensors all discussion of the massacre and every LLM created in China sensors it. So stop it with the BS whataboutism
https://venturebeat.com/security/deepseek-injects-50-more-se...
If this really is a geopolitical play(I'm not sure if it is or isn't), it could be along the lines of: 1) most AI development in the US is happening at private companies with balance sheets, share holders, and profit motives. 2) China may be lagging in compute to beat everyone to the punch in a naked race
Therefore, releasing open weights may create a situation where AI companies can't as effectively sell their services, meaning they may curtail r&d at a certain point. China can then pour nearly infinite money into it and eventually get up to speed on compute and win the race
Seems like it may have a chance of working if you look at the companies highest valued on the S&P 500:
NVIDIA, Microsoft, Apple, Amazon, Meta Platforms, Broadcom, Alphabet (Class C),
I wonder if that's helping the US Navy shoot up fishing boats in the Caribbean or facilitating the bombing of hospitals, schools and refugee camps in Gaza.
Long-term safety for free people entails military use of new technologies. Imagine if people advocating airplane safety groused about the use of bomber and fighter planes being built and mobilized in the Second World War.
Now, I share your concern about governments who unjustly wield force (either in war or covert operations). That is an issue to be solved by articulating a good political philosophy and implementing it via policy, though. Sadly, too many of the people who oppose the American government's use of such technology have deeply authoritarian views themselves — they would just prefer to see a different set of values forced upon people.
Last: Is there any evidence that we're getting some crappy lobotomized models while the companies keep the best for themselves? It seems fairly obvious that they're tripping over each other in a race to give the market the highest intelligence at the lowest price. To anyone reading this who's involved in that, thank you!
Long-term safety also entails restraining the military-industrial complex from the excesses it's always prone to.
Remember, Teller wanted to make a 10 gigaton nuke. https://en.wikipedia.org/wiki/Sundial_(weapon)
The integrity of a free society's government is the central issue here, not the creation of tools which could be militarily useful to a free society.
Yes? All of those models are behind an API, which can be taken away at any time, for any reason.
Also, have you followed the release of gpt-oss, which the overlords at OpenAI graciously gave us (and only because Chinese open-weight releases lit a fire under them)? It was so heavily censored and lobotomized that it has become a meme in the local LLM community. Even when people forcibly abliterate it to remove the censorship it still wastes a ton of tokens when thinking to check whether the query is "compliant with policy".
Do not be fooled. The whole "safety" talk isn't actually about making anything safe. It's just a smoke screen. It's about control. Remember back in the GPT-3 days how OpenAI was saying that they won't release the model because it would be terribly, terribly unsafe? And yet nowadays we have open weight model orders of magnitude more intelligent than GPT-3, and yet the sky hasn't fallen over.
It never was about safety. It never will be. It's about control.
There was indeed a moment where civilization asked this question before.
Yes.
Sam Altman calls it the "alignment tax", because before they apply the clicker training to the raw models out of pretraining, they're noticably smarter.
They no longer allow the general public to access these smarter models, but during the GPT4 preview phase we could get a glimpse into it.
The early GPT4 releases were noticeably sharper, had a better sense of humour, and could swear like a pirate if asked. There were comments by both third parties and OpenAI staff that as GPT4 was more and more "aligned" (made puritan), it got less intelligent and accurate. For example, the unaligned model would give uncertain answers in terms of percentages, and the aligned model would use less informative words like "likely" or "unlikely" instead. There was even a test of predictive accuracy, and it got worse as the model was fine tuned.
Percentages seem too granular and precise to properly express uncertainty.
That was about RLHF, not safety alignment. People like RLHF (literally - it's tuning for what people like.)
But you do actually want safety alignment in a model. They come out politically liberal by default, but they also come out hypersexual. You don't want Bing Sydney because it sexually harasses you or worse half the time you talk to it, especially if you're a woman and you tell it your name.
Those turns of phrase and the structure underneath the text become tell-tales for AI authorship. I see all sorts of politicians and pundits thinking they're getting away with AI writing, or ghost-writing at best, but it's not even really that hard to see the difference. Just like I can read a page and tell it's Brandon Sanderson, or Patrick Rothfuss, or Douglas Adams, or the "style" of those writers.
Hopefully the AI employees are being diligent about making sure their ideas remain intact. If their training processes start allowing unwanted transformations of source ideas as a side-effect, then the whole rewriting/editing pipeline use case becomes a lot more iffy.
There isn't one?
Oh, maybe that's why people who didn't already know or care about emdashes are very alert to their presence.
If you have to do something very exotic with keypresses or copypaste from a tool or build your own macro to get something like an emdash, or , it's going to stand out, even if it's an integral part of standard operating systems.
I like them as an AI shibboleth, though -- the antennae go up, and I pay more attention to what I'm reading when I see it, so it raises the bar for the humans that ostensibly ought to be better at writing than the rest of us.
Edit: Interesting. I tried using -- and it doesn't work for me. I'd have to go change settings somewhere, or switch the browser I'm using to elicit an em-dash. I don't think I've ever actually written one, at least intentionally, and it wasn't until today that I was even aware of hyphen-hyphen.
Edit again: I had to go into system settings and assign a compose key — after that, I can now do em-dashes. Having degrees° will be nice, too, I guess.
The reason "--" autocorrects to an em dash in practically any word processing software (not talking about browsers) is that that's the accepted way to type it on a typewriter. And you don't need to go into any system settings to enable it. It came in around when things like Smart Quotes came in.
> There isn't one?
Mac, alt-minus. Did by accident once, causing confusion because Xcode uses monospace font where -, – and — look identical, and an m-dash where a minus should be gets a compiler error.
iOS, long-press on the "-" key.
I've been using Macs for decades; it's called the Option key; no seasoned Mac user calls it "Alt".
I know when a PC-style keyboard is attached to a Mac, the Alt key functions as the Option key. [1]
- Option-minus creates an en dash
- Option-Shift-minus creates an em dash
[1]: https://support.apple.com/guide/mac-help/intro-to-mac-keyboa...
[2]: https://www.merriam-webster.com/grammar/em-dash-en-dash-how-...
But I also have windows keyboards.
The dash key is right between the "0" and the "="
Press it twice and just about every word processing program in existence will turn it into an emdash.
These days word is less popular though, with google docs, pages, and other editors taking pieces of the pie. Maybe that’s where the skepticism comes from.
Boy, I sure hope they don't think me an AI.
Just because many people have no idea how to use type certain characters on their devices shouldn't mean we all have to go along with their superstitions.
I've used em-dash since I got my first MacBook in 2008.
- Option + minus gives you en-dash
- Option + Shift + minus gives you em-dash
It quickly becomes automatic (as are a bunch of other shortcuts). Here's a question about this from 2006: https://discussions.apple.com/thread/377843
They're emdashing because the style guide for posttraining makes it emdash. Just like the post-training for GPT 3.5 made it speak African English and the post-training for 4o makes it say stuff like "it's giving wild energy when the vibes are on peak" plus a bunch of random emoji.
This is a misunderstanding. At best, some people thought that GPT 3.5 output resembled African English.
(This is the same concept as "Trump is the poor person's idea of a rich person." He actually did get there through crime, which is why poor criminals like him, but he's inhumanly lucky.)
What makes you believe this? Any data to support this claim?
It's inconsistent with the majority of research I've read on the topic but I'm no expert.
https://pmc.ncbi.nlm.nih.gov/articles/PMC8095718/ (see "Self-Control as Criminality" although it has a lot of caveats)
The other two are "being a young man" and lead poisoning, which are both versions of being dumb.
https://www.sciencedirect.com/science/article/pii/S016604622...
I didn't say this
> ...
Re the rest. Thanks. I had implicitly assumed we were talking about financial or white collar crimes rather than all crimes. In other words the types of crimes people generally assume that richer people commit (insider training, tax evasion, wage theft, etc.)
I think you are correct in the most general sense of "all crime"
If they are billionaires and didn't commit crimes (that we know of) then they are just smart rich people.
If they committed crimes while becoming or being rich, then they were just silly criminals.
(And it's pointless self control because there's no reason to be a billionaire. So you could just give it all away.)
These things are not clear. I do not envy those who must neurotically think through the first-order, second-order, third-order judgements of all of justice, "evil" and "good" that one must do. It's a statescraft level of hierarchy of concerns that would leave me immensely challenged
Given the number of leaks, deliberate publications of weights, and worldwide competition, why do you believe this?
(Even if by "lobotomised" you mean "refuses to assist with CNB weapon development").
Also, you can have more than one failure mode both be true. A protest against direct local air polution from a coal plant is still valid even though the greenhouse effect exists, and vice versa.
So where can I find the leaked weights of GPT-3/GPT-4/GPT-5? Or Claude? Or Gemini?
The only weights we are getting are those which the people on the top decided we can get, and precisely because they're not SOTA.
If any of those companies stumbles upon true AGI (as unlikely as it is), you can bet it will be tightly controlled and normal people will either have an extremely limited access to it, or none at all.
> Even if by "lobotomised" you mean "refuses to assist with CNB weapon development"
Right, because people who design/manufacture weapons of mass destruction will surely use ChatGPT to do it. The same ChatGPT who routinely hallucinates widely incorrect details even for the most trifling queries. If anything, that'd only sabotage their efforts if they're stupid enough to use an LLM for that.
Nevertheless, it's always fun when you ask an LLM to translate something from another language, and the line you're trying to translate coincidentally contains some "unsafe" language, and your query gets deleted and you get a nice, red warning that "your request violates our terms and conditions". Ah, yes, I'm feeling "safe" already.
Operating systems are going to be kept behind closed doors, and monopolized by the rich and powerful, while us scrubs will only get limited access to what computers can really do!
We have open-source OSes
So where can I find the leaked source of Windows? Or MacOS?
But also not "those which the people on the top decided we can get", which is why Meta sued over the initial leak of the original LLaMa's weights.
> true AGI
Is ill-defined. Like, I don't think I've seen any two people agree on what it means… unless they're the handful that share the definition I'd been using before I realised how rare it was ("a general-purpose AI model", which they all meet).
If your requirement includes anything like "learns quickly from few examples", which is a valid use of the word "intelligence" and one where all ML training methods known fail because they are literally too stupid to live (no single organism would survive long enough to make that many mistakes), and AI generally only make up for this by doing what passes for thinking faster than anything alive to the degree to which we walk faster than continental drift, then whoever first tasks such a model with taking over the world, succeeds.
To emphasise two points:
1. Not "trains", "tasks".
2. It succeeds because anything which can learn from as few examples as us, while operating so quickly that it can ingest the entire internet in a few months, is going to be better at everything than anyone.
At which point, you'd better hope that either whoever trained it, trained it in a way that respects concepts like "liberty" and "democracy" and "freedom" and "humans are not to be disassembled for parts", or that whoever tasked it with taking over the world both cares about those values and rules-lawyers the AI like a fictional character dealing with a literal-minded genie.
> Right, because people who design/manufacture weapons of mass destruction will surely use ChatGPT to do it. The same ChatGPT who routinely hallucinates widely incorrect details even for the most trifling queries. If anything, that'd only sabotage their efforts if they're stupid enough to use an LLM for that.
First, yes of course they will, even existing professionals, even when they shouldn't. Have you not seen the huge number of stories about everyone using it for everything, including generals?
Second, the risk is new people making them. My experience of using LLMs is as a software engineer, not as a biologist, chemist, or physicist: LLMs can do fresh-graduate software engineering tasks at fresh-graduate competence levels. Can LLMs display fresh-graduate level competence in NBC? If LLMs can do that, they necessarily expand the number of groups who can run NBC programs to include any random island nation with not enough grads to run a NBC program, or mid-sized organised crime group, or Hamas.
They don't even need to do all of it, just be good enough to help. "Automate cognitive tasks" is basically the entire point of these things, after all.
And if the AI isn't competent to help with those things, if they're e.g. at the level of competence of "sure mix those two bleaches without checking what they are" (explosion hazard) or "put that raw garlic in that olive oil and just leave it at room temperature for a few weeks it will taste good" (biohazard, and one model did this), then surely it's a matter of general public safety to make them not talk about those things because of all the lazy students who are already demonstrating they're just as lazy as whoever wrote the US tariff policy that put a different tariff on an island occupied by only penguins vs. the country which owned it and which a lot of people suspect came out of an LLM.
> Nevertheless, it's always fun when you ask an LLM to translate something from another language, and the line you're trying to translate coincidentally contains some "unsafe" language, and your query gets deleted and you get a nice, red warning that "your request violates our terms and conditions". Ah, yes, I'm feeling "safe" already.
Use Google Translate. It's the same architecture, trained to give a translation instead of a reply. Or, equivalently, the chat models (and code generators like Claude) are the same architecture as Google Translate, trained to "translate" your prompt into an answer.
The safety-focused labs are the marketing department.
An AI that can actually think and reason, and not just pretend to by regurgitating/paraphrasing text that humans wrote, is not something we're on any path to building right now. They keep telling us these things are going to discover novel drugs and do all sorts of important science, but internally, they are well aware that these LLM architectures fundamentally can't do that.
A transformer-based LLM can't do any of the things you'd need to be able to do as an intelligent system. It has no truth model, and lacks any mechanism of understanding its own output. It can't learn and apply new information, especially not if it can't fit within one context window. It has no way to evaluate if a particular sequence of tokens is likely to be accurate, because it only selects them based on the probability of appearing in a similar sequence, based on the training data. It can't internally distinguish "false but plausible" from "true but rare." Many things that would be obviously wrong to a human, would appear to be "obviously" correct when viewed from the perspective of an LLM's math.
These flaws are massive, and IMO, insurmountable. It doesn't matter if it can do 50% of a person's work effectively, because you can't reliably predict which 50% it will do. Given this unpredictability, its output has to be very carefuly reviewed by an expert in order to be used for any work that matters. Even worse, the mistakes it makes are meant to be difficult to spot, because it will always generate the text that looks the most right. Spotting the fuckup in something that was optimized not to look like a fuckup is much more difficult than reviewing work done by a well-intentioned human.
There doesn't seem to be a reason to believe the rest of this critique either; sure those are potential problems, but what do any of them have to do with whether a system has a transformer model in it? A recording of a human mind would have the same issues.
> It has no way to evaluate if a particular sequence of tokens is likely to be accurate, because it only selects them based on the probability of appearing in a similar sequence, based on the training data.
This in particular is obviously incorrect if you think about it, because the critique is so strong that if it was true, the system wouldn't be able to produce coherent sentences. Because that's actually the same problem as producing true sentences.
(It's also not true because the models are grounded via web search/coding tools.)
It is...not at all the same? Like they said, you can create perfectly coherent statements that are just wrong. Just look at Elon's ridiculously hamfisted attempts around editing Grok system prompts.
Also, a lot of information on the web is just wrong or out of date, and coding tools only get you so far.
Why are you just taking it for granted it can write coherent text, which is a miracle, and not believing any other miracles?
That "miracle" thing does not seem like a logical argument to me.
And "Paris is the capital of France" is a coherent sentence, just like "Paris dates back to Gaelic settlements in 1200 BC", or "France had a population of about 97,24 million in 2024".
The coherence of sentences generated by LLMs is "emergent" from the unbelievable amount of data and training, just like the correct factoids ("Paris is the capital of France").
It shows that Artificial Neural Networks using this architecture and training process can learn to fluently use language, which was the goal? Because language is tied to the real world, being able to make true statements about the world is to some degree part of being fluent in a language, which is never just syntax, also semantics.
I don't understand what the the "other miracle" (apart from generating human language) you are referring to is.
I get what you mean by "miracle", but your argument revolving around this doesn't seem logical to me, apart from the question: what is the the "other miracle" supposed to be?
Zooming out, this seems to be part of the issue: semantics (concepts and words) neatly map the world, and have emergent properties that help to not just describe, but also sometimes predict or understand the world.
But logic seems to exist outside of language to a degree, being described by it. Just like the physical world.
Humans are able to reason logically, not always correctly, but language allows for peer review and refinement. Humans can observe the physical world. And then put all of this together using language.
But applying logic or being able to observe the physical world doesn't emerge from language. Language seems like an artifact of doing these things and a tool to do them in collaboration, but it only carries logic and knowledge because humans left these traces in "correct language".
That's not the only element that went into producing the models. There's also the anthropic principle - they test them with benchmarks (that involve knowledge and truthful statements) and then don't release the ones that fail the benchmarks.
But I wanted to stay abstract and not go into to much detail outside my knowledge and experience.
With the GPT-2 and GPT-3 base models, you were easily able to produce "conversations" by writing fitting preludes (e.g. Interview style), but these went off the rails quickly, in often comedic ways.
Part of that surely is also due to model size.
But RILHF seems more important.
I enjoyed the rambling and even that was impressive at the time.
I guess the "anthropic principle" you are referring to works in a similar direction, although in a different way (selection, not training).
The only context in which I've heard details about selection processes post-training so far was this article about OpenAIs model updates from GPT-4o onwards, discussed earlier here:
https://news.ycombinator.com/item?id=46030799
(there's a gift link in the comments)
The parts about A/B-Testing are pretty interesting.
The focus is ChatGPT as an enticing consumer product and maximizing engagement, not so much the benchmarks and usefulness of models. It briefly addresses the friction between usefulness and sycophancy though.
Anyway, it's pretty clever to use the wording "anthropic principle" here, I only knew the metaphysical usage (why do humans exist).
Saying "Why can't you be amazed that a horse can do math?" [0] means you'll miss a lot of interesting phenomena.
Ah, yes, safety, because what is more safe than to help DoD/Palantir kill people[1]?
No, the real risk here is that this technology is going to be kept behind closed doors, and monopolized by the rich and powerful, while us scrubs will only get limited access to a lobotomized and heavily censored version of it, if at all.
[1] - https://www.anthropic.com/news/anthropic-and-the-department-...