If it weren’t OAI and Anthropic, it would be others.

If it weren’t US companies, it would be others.

It is far too useful now to stop large scale deployment, it isn’t a matter of specific people or countries not being cautious or prudent enough. The entirety of the Earth is at issue now and there will always be people who cause the tragedy of the commons (if indeed AI turns out to be harmful in the long run, which is not remotely certain).

Look at what happened with fossil fuels.

There is something deeply rotten about the US mindset though, them being American is definitely a problem.

> Look at what happened with fossil fuels.

The west had a 100 years head start, on top of that these countries are producing everything you consume, don't start pointing fingers. Chinese current co2 per capita is the same as the US in 1900

Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did, and the Chinese followed in.

Had AI been invented in the 20th century, it would have been kept secret and used only by state actors to facilitate genocides at mass scales. But we live in different times.

Uh? In what universe are openai and anthropic not shadow groups?

Oh, how nice of them, such charitable souls...

But if you're an angry hacker in full on culture war witch hunter mode, then of course nobody can say something that's even slightly different than full-on hate without you interpreting that as aligning with the evil witch.

How could modern AI have helped those groups you mention in their endeavour? Really fucking easy and effectively, and you don't need a lot of imagination to think of how. From surveillance and interrogation to pinpoint determination of ethnicity and much more. Not to mention translation between human languages.

They literally targeted people for their past opinions, how they looked, etc. which you can now determine and process at break neck speed, you can even remove humans from the loop and have the machine determine who's an undesirable or not.

How is this a serious question lol?

What is that “something”?

The world’s biggest philanthropists are American.

The US has for decades invested into charities and human development around the worlds .

> deregulation at all cost

The US has regulations. The Constitution of the United States is the oldest and longest-standing written and codified national constitution in force in the world.

> Christian brained politicians

We have separation of church and state.

The Sackler family of opioid infamy were huge philanthropists. Not saying that the AI companies are like Purdue Pharma, but clearly philanthropy does not compensate for societal damage.

The data does not lie.

Yeah because it includes lobbies and turbo billionaire pastors from texas using "charity" money to buy rolls royces as well as other highly retarded """charities""" that aren't even legal in the rest of the developed world

You have senators and wannabe senators quoting the bible on TV as their guiding star as we speak right now, give me a break

On top of that you can separate the church and the state all you want, when people like Thiel show up it doesn't matter, these people are deeply troubled, if they were muslim and saying 1/10th of what he says you'd send them to guantanamo

Doing what? Rushing to make trillion dollar companies, rather then developing the tech and their companies in a sustainable way? I don't even know what in the original comment this would be a response to.

The particular form of the pursuit of power practiced by OAI and Antrophic - the creation of trillion dollar companies, is one of the most benign in human history. It's so unnaturally benign, that I doubt it's going to survive for much longer.

Political power is when people act in concert, when leaders have people behind them because those people agree with how they're being lead etc., it's not something that happens unilaterally. You can physically force people, but for what when you can have robots that you don't have to force?

So that kind of leaves wiping out nearly all humans, for a few hundred thousand remaining people to get served by robots and play golf. Which I would say is what we're kinda on track for. But that wouldn't be a race between nations, that race would just be pretend to get the tech online; it would rather be a race between all humans and a bunch of sick people.

AI doesn't have to be used offensively to give a nation power. The machines can already design new software at the drop of the hat, it will at some point become capable of designing new consumer products and industrial products. Putting a tireless intelligence at the task of looking for new optimizations in your industrial processes and factories will eventually give you economic dominance, one way or another.

Offensive uses don't need to be as extreme as you imagine them. We're already starting to see the effects of having meticulous automated hackers.

Finally, political power itself doesn't need to come from people acting in concert. There's always been wheeling and dealing and intrigue and bureaucracy. Once AI is sufficiently versed in legal matters, it'd allow political interest groups to overwhelm governments and private enterprises with objections and appeals, paralyzing their ability to carry out their agenda until they make concessions.

The possibilities of using intelligence for acquiring more power are literally endless..

That doesn't mean that everyone has to use to use it.

We're only starting to see the beginning of the negative impact of AI to the job market and society. If if turns out that allowing most jobs to be replaced by AI is bad for society and the economy (super high unemployment - nobody buying more then necessities), then wouldn't it be smarter to not do that?

If that's the case, then each country can decide for themselves.

Dario's "Machines of loving grace" UBI/foodstamp "utopia" sure doesn't sound like a recipe for success.

Value is simply the mutual understanding that something should cost a certain amount, and the system we've built emphasizes the need for profit above all else. When a set of companies comes along with a new, impressive technology claiming that they can automate away the white collar workforce, they're trying to capture as much value as they can.

It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.

I don't agree that "I am just a market actor behaving rationally" somehow absolves you of any moral duty

I'm not an ethicist or anything but I have studied quite a lot of ethical frameworks and I've never encountered one that I think would agree with this

I didn’t expect a full throated defense of mugging and burglary on HN this morning, but here we are.

Or have you forgotten that Sam Altman was ousted by individuals that shared your concerns, and it was the employees that rebelled and wanted Altman back?

That HN post was infamous and nearly took down this site.

The AI companies are as much being pushed along by investors as they are in the driving seat, but the push is in the direction of getting real products in front of people and experimenting with this miraculous tech everywhere it could possible work.

How many, approximately?

Ignoring the implementation details, if we treat the whole global technology world as one single AI system made up of AI companies, independent developers/researchers, and end users, what would be the training goal and evaluation?

I feel like you’re not acknowledging the inevitability of the technology being developed further. You’re also ignoring competition and the basic fact that people do research and push the boundaries of their field naturally.

If the major frontier AI labs had decided, as you said, to make a fancy autocomplete and a better Grammarly and then simply stop and coast, do you really believe that every AI researcher and entrepreneur in the entire world would look at the situation and decide that the status quo is where it should end?

Humans don’t work that way. People would get ideas and want to push the limit. If the frontier labs stopped developing their technology, other companies would be created to carry on. Academic researchers would push the limits further. It would be discovered that the AI could go further and we’d get back to this same eventuality with a different set of people leading it.

I also disagree with your characterization of this as a get rich quick scheme for AI leaders who want to make a lot of money and then be done. I think this characterization feels truthy if your only conception of startups is as a gamble to retire early, but the people who build and run these companies actually do like building and running companies. If the AI leaders and executive management you’re talking about were separated from their jobs today, even with billions of equity, most of them would be out starting new companies next month because that’s what people like that do. I’m not saying this to “glaze” them or whatever I’ll surely be accused of in the comments, I’m stating some facts about this space. You don’t need to look far to find examples of this already happening.

This looks a lot like the “if I didn’t do $badthing someone else would have” line of reasoning.

With AI, the point is who uses it for what.

Think about it like a magic wand that makes your wishes come true. Would you really want everybody to have one? Who should and who shouldn't? Who gets to decide?

Humanity's strategy so far has been to pull marbles out of the bag as fast as we can. And that's been working out, because most marbles are white. But we're insanely lucky we haven't had a nuclear war. The USSR's early warning systems picked up 5 incoming american missiles in 1983. They didn't fire back because one of the officers on shift didn't believe the reports (why only 5 missiles?), and he declared the reports were fake. If a different person was on shift, there's a good chance nuclear war would have broken out because of a sensor glitch. And that's far from the only instance.

And now we're improving AI capabilities as fast as we can, despite it being pretty clearly another grey ball. AI has a lot of amazing uses. But why are we rushing so much? Doesn't seem wise.

https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar...

> But why are we rushing so much?

Talking about these things with elaborate analogies only obscuring the point you already stumbled into: Humanity advances because humans are curious, driven, like to discover things, and like to push the boundaries.

Talking about humanity like one entity that operates together misses the point: There are billions of people in the world and some of them are going to discover new things.

There is no general counsel where humanity gets together and decides what humanity works on. There is no singular bag with a singular “humanity” pulling marbles out one at a time.

The analogy is too broken to be useful for the discussion.

This is true, but not all the time, and transparently not the case for the current leaders in AI development. Their motivator is money.

There are plenty of examples of this, e.g. when we discovered how to harness nuclear energy, we quickly realized that extreme measures needed to be taken to ensure we don’t collectively destroy ourselves.

You’re speaking of “we” as if the whole world was operating in unison, while there’s a war going on in Iran over nuclear disagreements.

The nuclear situation only reached stability after multiple countries fully developed it. I don’t see the parallels to halting AI.

Nuclear Non-Proliferation Treaty (1968)

Biological Weapons Convention (1972)

Asilomar Conference on Recombinant DNA (1975)

We've done this sort of thing plenty of times around specific, dangerous technologies.

Also the bio/DNA ones are quickly falling apart as the technology to do these kinds of things on a small scale are readily available. To the point where we're finding illegal wet labs in the US.

Nuclear was more of an exception because of many of difficulties in purifying stuff for weapons and its detectability.

AI is a much, much, much harder problem is it uses the already existing infrastructure as a base. Unless we regulate, monitor, and ban all private usage of technology AI can be developed at a small scale.

If there is a room full of people, and one person gets a gun, who would you pick to have it?

Yourself, obviously.

Maybe america will end up the only world power holding the secrets of LLMs? ... Oh no...

If nuclear weapons were as easy to build as AI models are to run, we would have all been dead decades ago.

There might have been. But the notes from the Manhattan project were leaked almost immediately to the russians by informants.

How is any of that good?

---

Simply put humans are at risk of any power scaling. We exist safely in a very tiny temperature and energy environment and the rest of the processes in the universe operate widely out of that strict safe area.

It's also a catch-22. There are large rocks in space that will eventually end us unless we do something about that. But having the power to do something about that gives us the same power to also end everyone on the earth.

> Nazi Germany undertook several research programs relating to nuclear technology, including nuclear weapons and nuclear reactors, before and during World War II. [...] The first effort started in April 1939, just months after the discovery of nuclear fission in Berlin in December 1938, but ended shortly ahead of the September 1939 German invasion of Poland, for which many German physicists were drafted into the Wehrmacht. A second effort under the administrative purview of the Wehrmacht's Heereswaffenamt began on September 1, 1939, the day of the invasion of Poland. The program eventually expanded into three main efforts: Uranmaschine (nuclear reactor) development, uranium and heavy water production, and uranium isotope separation. Eventually, the German military determined that nuclear fission would not contribute significantly to the war, and in January 1942 the Heereswaffenamt turned the program over to the Reich Research Council (Reichsforschungsrat) while continuing to fund the activity.

> The program was split up among nine major institutes where the directors dominated research and set their own objectives. Subsequently, the number of scientists working on applied nuclear fission began to diminish as many researchers applied their talents to more pressing wartime demands.

> [...] Heisenberg said that in 1939, the physicists at this second meeting concluded that "in principle atomic bombs could be made, but they also emphasized that it would take a number of years",[24] certainly not less than five. He also said that he "didn't report it to the Führer until two weeks later and very casually because I did not want the Führer to get so interested that he would order great efforts immediately to make the atomic bomb. Speer felt it was better that the whole thing should be dropped and the Führer also reacted that way."

> [...] Speer states that the project to develop the atom bomb was scuttled in the autumn of 1942. Though the scientific solution was there, it would have taken all of Germany's production resources to produce a bomb, and then no sooner than 1947.[39] Development did continue with a "uranium motor" for the navy and development of a German cyclotron. However, by the summer of 1943, Speer released the remaining 1200 metric tons of uranium stock for the production of solid-core ammunition.

There are ways of calling this "weren't working on a nuclear bomb", but it seems to me like "they wanted to, were moving in that direction, and would have done it if they had the resources and weren't distracted by shorter-term considerations".

That sounds very much like "they weren't really working on it". They explored the options and decided not to.

Whatever you want to call it, that's what I was referring to.

Like all forms of evil once invited into your house it does not leave at your say-so, another profoundly negative feature of nuclear warfare is that now we've invented it unilateral disarmament will likely always be strategically impossible.

Still seems a moral cop-out, and while true, the timeline would certainly have changed, if not the inevitability, if key figures such as himself had opted out.

I'm not arguing about the consequences or responsibility. I'm asking which humans would be capable of stopping the development of AI at this point in history.

If we're talking game theory, a better model is the Stag Hunt [1]. At some point the equilibrium strategy is to cooperate.

[1]: https://en.wikipedia.org/wiki/Stag_hunt

It’s difficult to understand where this all is coming from because it’s so strangely disconnected from history. It’s not even consistent with current world events. The US is in a war with Iran over nuclear proliferation while also worrying about North Korea’s nuclear program. China and the US are in a trade war and Taiwan is under threat due to semiconductor capabilities (among other things, obviously). Open weight models allow anyone in any country to do AI work on their laptop. The top story on HN yesterday was a small team of academics making some cool discoveries on improving LLMs without frontier lab level compute.

How anyone can see all of this, in addition to decades of history of attempted nuclear diplomacy and wars, and posit that the whole problem is easily solved by simple agreements is confusing to me. The confidence with which these alternatives are being proposed and the unwillingness to acknowledge second order effects (like AI researchers moving to a different country and continuing to work) is baffling.

But yeah, there are a lot of comments from genie-rebottlers that are not very grounded in reality. They're functionally equivalent to saying "hey, everyone in the world, let's not actually use electricity after all" in 1905.

Despite this, when discussing the egregious behavior of the frontier labs, there is vanishingly little acceptance of the idea that 1) yes, it probably was inevitable in some ways AND 2) those involved, particularly at the top, are still responsible for the consequences of their (bad) decisions.

1. Is it both inevitable and still they are responsible? 2. Or it was inevitable and they were just an instrument to make it happen. 3. Or it wasn't inevitable and if it were not for some specific people then it would not have happened?

I am leaning more towards 2. But if that is the case why are only specific people reaping the benefits of this?

If it is 3 or 1, then why aren't we holding them accountable?

So yes, they are 100% responsible for damage they caused and did not had to happen. And no, "I had no choice, because it was the only way to become billionaire" is not an excuse.

They were NOT "instrument". No one manipulated them. No one forced them. Nothing forced them. They themselves wanted to and did not cared

People are starting to panic, and few things bring out full blown idiocy and disregard for ground truth faster than widespread panic.

You must be joking. Altman is a sociopath, Amodei is either too stupid or too evil to stop working on tech he claims will have a 10% chance of destroying humanity (and in either case he should be imprisoned to protect the rest of us from him), and Musk is a man-child who will lash out at anyone and everything who dares to speak ill of him. There's absolutely nothing "lucky" about having these jokers in charge of stuff.

It's similarly easy to come up with "what Altman should do better". The dude is a genuine sociopath in charge of a company trying to make everyone's lives worse off in order to enrich him and his investors. It doesn't get much worse behaved than that.

So yeah, I'm negative against their development of AI along with the general public, and for good reason it would seem.

Also, I don't want super intelligence. I want humans to remain in control. Why are they trying to build something to replace humanity? Why don't we got a vote in this? All 8 billion of us not developing frontier models with the expressed goal of achieving AGI followed closely by ASI.

It's a pity because as computer scientists this should have been such a fun technological moment for us. What a cool thing to experiment and explore new open doors with. But instead it was immediately the driver of mass job loss, societal upheaval, and a double digit percent chance of ending humanity, espoused by the AI lords themselves.

The list of possible reasons is pretty short and mostly not really good for most of us, as far as I see it

I do. Quite a lot of people do. And that is because we believe it will lead to a better life for most humans.

-G. Hinton

The models we currently have aren't going away, and especially the ones small enough to run at home will continue to be used, but if these data center investments end up bankrupting Big Tech, I don't think we're ever going to see anything beyond that. Not for a long time, at least.

My guess would be that once the money runs out, things slow down, but incremental progress continues to be made. A focus on smaller models will improve their capabilities. We’ll get better at driving them. It’ll be used in places where it truly makes sense, instead of anywhere & everywhere just because it’s easy.

I don’t actually know what the financials are, but even without the effective subsidy from VC money, I have to believe there are things AI’s doing today that’s way cheaper than what it would cost to have a human doing equivalent work. And that dynamic will drive further innovation in the space for a long time.

What does this actually mean? In what sense are people using this term. In the most strictly literal sense then sure, nothing is inevitable, because all paths are open.

But in the practical sense, I am quite happy to use the term inevitable for many technologies. I think if ASI is possible (and I think it is), then it is also inevitable. I would happily work towards developing ASI technology, because I believe it will be ultimately good for the world. But it sounds like a lot of people want to lock me up or even kill me if I did that.

AI indeed goes far beyond "autocomplete" and other nonsense. But the question is, whether it's a good idea to just let the "market" run wild with it.

Short-term capital interests don't align with the public good. AI greatly scales capability, but without anything to restrain it, that implies those non-aligned interests overwhelming society's defenses, outright destroying it in the process.

The idea of rich people being some kind of benevolent caretakers, who independently and dependably foresee and manage the consequences of their actions in everybody's best interest, is laughably wrong already.

With AI, that reality gap will swallow whatever is left.

Without capitalism, you wouldn't be able to restrain AI adoption any more than the armored knights of the middle ages were able to restrain the adoption of the firearm.

I think the vast majority of the rich are greedy bastards, but IMO being rich does not imply greedy bastard 100% of the time.

This doesn't seem right: it doesn't match the history of reality, in which capitalism has repeatedly stymied rather than fostered competition.

The argument from nature is a fallacy. Just because something is doesn't mean it ought to be.

I operate within the logic of how the world actually works.

Which I find infinitely more useful for practical topics like the reality of AI development in a global economy.

Fantasizing about alternative realities where all the leaders of the world unite and halt technology development without any of them secretly defecting and developing it further isn’t very useful to discuss. These things aren’t going to happen.

it has kinda worked for nuclear weapons

That’s the inverse of what I said.

AI has no such hurdles.

Transitioning from the current "order of things" to one compatible with the existence of things like AI is a necessity, not a "fantasy".

When you imagine people as savages to be outdone in their savagery, you become one yourself. Spend more effort on finding ways not to.

Ignorance of what? All of the previous successful attempts where the entire world reorganized itself into a single world government?

> Transitioning from the current "order of things" to one compatible with the existence of things like AI is a necessity, not a "fantasy".

I think you’re putting the cart before the horse. You’re assuming it’s necessary and then making another leap to assume that being necessary means it must also be possible.

Additionally, you don't need all the leaders of the world to agree to anything, you only need US and China.

Additionally x2, if you agree with the common stance that China makes the vast majority of their progress by distilling US models, you don't even need them to agree to anything.

You're really suggesting the following bans. A ban on computer science and algorithms. A ban on any device that can play games that aren't encrypted or DRM locked from an approval authority. A ban on any kind of unregulated computing without government approval and licensing.

>China makes the vast majority of their progress by distilling US models

Eh, this is a budgetary decision, not a technological one. Why spend billions when the US can spend billions and you can spend millions. China more than has the power to do what we're doing if they chose to do so, their biggest limitation is speed of making new compute, not money or brains.

You know what, fuck it, that's exactly what I'm suggesting. Let's ban personal computers. The only consumer hardware allowed should be a HTTPS client directly burned on the chip that can only connect to government servers. Absolute all timer of a straw man. This is probably the last comment I'm writing on this site, this place has become unsufferable.

Believe it or not, there already are things you're not allowed to do with computers that you otherwise own, and we have ways of enforcing them other than trusted computing -- the threat of putting you in a cell for a few decades usually does the job. That's without getting into the fact that you're not going to train 1T-sized models on your laptop, and billion dollar GPU farms aren't precisely hard to keep track of.

I'm not rooting for AI companies or hustlers that are out to ride the wave and get rich.

However your claim is just hyperbole. I've used it in cross-cutting projects to extend protocol on desktop, iot device, which involves holding premise long enough to co-develop and test. It is also fantastic in debugging multi-threaded systems.

Some times it comes from people who try out ChatGPT free tier or the Google AI summary and think it represents the state of the art.

More recently I suspect it’s from people who build their opinion based on being contrarian and cynical: They dislike the AI people and distrust everything they say. If the AI people say it’s changing the world, then the cynical and contrarian belief is to assume that the technology is actually very bad and they’re just lying to make a lot of money and retire.

These takes were more understandable in the earlier phases of AI when it really was very limited, but it’s getting harder to understand how someone can be thinking of AI as just a slightly better autocomplete or slightly better Grammarly at the end of 2026.

That wasn't the point. My point was that if you had rolled LLMs out like that, as embedded technology in existing products. Slowly, over time, you'd have a lot less cynical people. I think the hostility towards AI product in large parts comes from the disruption and glancing over the negative parts. What we're doing now is the equivalent of putting millions of cars on roads which five years ago had never seem a car, only horses. We're doing so without any understanding of how that might affect society. Social media was a similar story. It was slower, yet we still didn't catch the majority of the negative consequences.

If it turns out that the people behind social media, and smart phones, don't care about e.g. people mental well being, then why would I trust that the people behind AI wouldn't behave the same way?

1. enhanced autocomplete -- either line (e.g. JetBrains small language models) or longer predictions;

2. chat -- asking questions related to the task you are doing;

3. tool assisted/agentic -- letting models interact with tools to perform tasks.

I personally find 1 & 2 helpful for my development tasks. For 3, I'm cautious of using it for several reasons:

1. lack of control of what gets sent to the model (e.g. if you have confidential user data, proprietary code and/or data, access tokens/private keys);

2. letting the models run arbitrary commands (modifying the system, deleting files, etc.);

3. not having control over the design process, formatting of the files, etc.

Presenting AI as something it isn't doesn't change the reality which is like the Wait But Why thing https://waitbutwhy.com/2015/01/artificial-intelligence-revol...

That was written in 2015 before LLMs were much of a thing and has been fairly spot on so far. This is not about LLMs.

Aside from the AI dungeon app, this was my first real exposure to LLMs wrt github's copilot extension. It sucked, and did a lot of stuff wrong, but they got exactly 5 bucks from me to try it.

It's sorta crazy to think how many years ago that was now.

Then the Chinese came along and showed that you can take those model weights, repackage them as open weights, and undercut the business model. Suddenly Dario et al. start talking about how dangerous AI is and how it needs to be regulated right now, before someone's abliterated open-weight Chinese-provided business-undercutting model destroys the world. They want their walled gardens regulated into being the only game available.

Both stages of the game—raising capital and regulatory capture—require hyping up the disruptive aspect of AI.

They certainly don't care when a model hallucinates or an agent does something it shouldn't be doing. It's your fault and they charge you for the tokens.

The only leader I’m not aware of really taking a more active role in safety on the daily is Elon Musk, so I’d like to see evidence supporting what he is doing or not doing on that front if someone from SpaceX can share their experience. The evidence on the environmental front from SpaceX is very concerning and is disregarding the wellbeing of the world and local populations in my opinion.

That’s why the AI heads all have extremely dramatic narratives. They are the end product of a selection tournament for raising massive amounts of money pre-product.

You make it sound like this was a feasible option. This technology would exist regardless of how any one company decides to frame it. If OpenAI decided to go with a more conservative approach rather than just throwing ChatGPT out for the masses, then all it would take is for one person to realize they didn't take full advantage of this technology to just make their own ChatGPT.

>The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.

You can criticize the current tech leaders all you want, but what you're describing is a fundamental dynamic of capitalism. You must choose the most profitable option, otherwise you have chosen the wrong option. If the current tech billionaires all vanished just as ChatGPT was released, they'd be quickly and easily replaced with roughly equivalent individuals who would effectively operate the same exact way. That's true because they're operating based on the mostly deterministic policy of profit maximization. They don't really get a say in it; they must follow the path towards maximizing expected future returns, otherwise they'll get canned by the investors who are seeking maximized expected future returns.

None of this relieves them of their personal ethical responsibilities either. I'm just pointing out that your comment might as well be describing an alternate reality where everyone lives peacefully with each other and nobody goes without what society can afford to offer them. It's equally as unrealistic.

Can someone who thinks this about Altman explain to me why he has repeatedly declined to take any equity in OpenAI? I genuinely want to understand your theory of mind here. Do you think he's actually in it for the power, and for some reason thought taking equity would undermine his effort at accumulating power? It just makes no sense to me.

It seems obvious that this same dynamic is in play with the frontier AI companies now. They know if they win the race to the they'll be forgiven their sins, or anyway they'll be beyond the reach of effective punishment.

https://en.wikipedia.org/wiki/Worse_is_better