Other commentors have made good points that these guardrails are counter productive for well intentioned cyber security, because I can't use it to test and harden my own software.
Anthropic guardrails seem to be more about protecting their business (distillation), than they are about public safety.
[1] https://www.anthropic.com/news/detecting-and-preventing-dist...
If Google starts calling ads “Best Links” that doesn’t make it correct nor canonical; the correct term is still ads.
Traditionally, distillation is when you get the actual logits of a model response (not exposed via API for years) and then use that to train a model.
Anthropic is offering a commodity product and trying to convince you it isn’t.
It’s even in the name, it’s a myth and a fable. Never happened doesn’t exist.
Also I believe at least on coding that qwen is now the frontier model, fable is its copy of frontier models. In the same way that the Ferrari Luce is an expensive imitation of a SU7 Ultra.
The delusions people live in just to be a hater.
At least the Chinese have the decency of giving back the model weights and not put BS censorship because “it’s too dangerous”.
the problem is so large scale that distill attempts attribute to a decent share of their token revenue generally.
The answer is, the organization making the powerful tool. The people in charge of Anthropic.
Not only that, but they've also written at length about exactly what their opinions and values are: https://darioamodei.com/
You may not agree with the decisions that they make, but they're hardly mysterious. Not something to wonder about.
Again, just because someone has values, doesn't mean they have values you think are good.
What a disgusting country filled with hollow degenerates. No wonder you keep voting for a senile grifting pedophile.
As for making money, you're right, it's not one of your values. It takes a special case of main character syndrome to think it's not anyone's value.
This whole business just keeps getting dumber.
1: https://darioamodei.com/post/policy-on-the-ai-exponential
Frontier AI models, like airplanes, should
be required to go through technical testing
and auditing, and their release should be
blocked or reversed as a threat to public
safety if they do not meet high standards
of safety. I am grateful to see the Trump
administration’s Executive Order move
incrementally towards a greater role for
government in AI, though Anthropic’s proposal
recommends even further action.
First sentence by itself is mundane "regulators are good", which most people agree with, and also libertarians will object to regardless of leader.
Second sentence is obviously sucking up, though is the same level of sucking up found on every stereotypical LinkedIn post.
(Unless they are piping the F1 Mercedes theme song in the announce system at anthropic, in which case maybe you are right)
But I tend to agree, just saying it's a "pretty reasonable statement" and leaving it at that is beyond the pale for anyone who doesn't have an undisclosed stake in the argument.
That is "pretty reasonable" to most people (except the tech-libertarian crowd).
Or maybe he has. I don't know.
I don't really agree with their point here, but there are plenty of people in the AI community whose views are aligned with Anthropic's. That doesn't make them shills.
It's actually important those views are put forward.
A place like LessWrong has the opposite problem - there is no one there who questions the "safety narrative" so the discussion swings more and more towards the extreme end of that spectrum.
My concern is that they won't be pointless in effect. Make no mistake: if Amodei has his way, possession of unvetted model weights will be treated like possession of CSAM is today. And at the same time Amodei calls for that, others are calling for the deployment of technical measures that will make it easier to enforce such laws.
All to the sound of thunderous applause on "Hacker News."
From that paragraph?
Even granting it is sucking up, that is not replacing.
If you think this is OK, I'm not sure what led you to a site called "Hacker News," but fortunately there are plenty of others.
Not sure who you are arguing with really. There seems to be a few logical leaps in between each response. I also didn't say anything like that.
Just before asking for approval to run, it said one thing it wanted to "flag before running" was "Rate-limit and auth testing against prod will generate some 4xx noise in Railway logs and could trip the form rate limiter — harmless, but saying it now."
Ok fine, I said go for it, and it says:
"Running it. Quick recon first (prod URLs + the prior-findings baseline), then I'll fan out the audit tracks with adversarial verification."
Immediately after, I got the Fable warning about how it can't continue because of safety concerns, switching to Opus. In the end, Opus did a good job thanks to whatever Fable suggested doing. Things were fixed that Opus missed in a security/performance audit just the week prior. But what surprised me is that it used 55 agents. Burned 80% of my 5-hour window in 15 minutes (5x Max plan). I've never had Opus do that before on these audits.
I can sympathize with the argument for the cyber refusals - especially as a temporary measure - especially if Mythos is available to those trying to defend against vulnerabilities.
The LLM development nerfing (and now refusals) is very different though. Anthropic has even said it isn't just for safety reasons:
> Using Claude to develop competing models already violates our Terms of Service, but enforcing this restriction through our safeguards avoids accelerating the actors most willing to violate these terms.
It's at least partially an anti-competitive measure.
The closest analogy is putting measures in a compiler to stop it being able to build other compilers.
Another analogy is priesthoods with secret religious knowledge that "only they are qualified to know".
“The request could assist the development of competing AI models, which is restricted under Anthropic's commercial terms. Benign machine learning work can also trigger this category.”
Source: https://platform.claude.com/docs/en/build-with-claude/refusa...
You’re buying into the hype they’re trying to create here.
Imagine your healthcare provider just sometimes decided not to read your test results very carefully and you risked death? Now realize that healthcare providers use Claude now and that scenario wasn't hypothetical.
Ah "Mr. Monty Carlo", it says here that you have a UTI, we'll get those kidneys removed ASAP so that won't happen again.
I think it's a fundamentally impossible thing to fix, though. There's no 100% correct answer.
That said, this thing is in real production use with war fighters, doctors, and financial experts. Just YOLO'ing to a dumber model midway through a multi-step process and pretending everything is fine is not a real or defensible option. Someone is going to die, and its going to be the fault of whoever decided to make this the default rather than opt-in.
Personally, I couldn't live with myself.
Only in the same sense that Standard Oil considered themselves the stewards of petroleum. There's benefit of the doubt and then there's just fanfiction. Do not forget that this most aggressive "guardrail" of theirs was not for any safety reason, but just to stop other labs from catching up to their product. They care less about hindering bioweapons, malware, and hate speech than they do free market competition.
No, it's not because it doesn't exist (yet) and its further from reach than the other examples. Also, the guardrails are also framed as restricting usage for the development of "competing" products/services.
In isolation it's not, but I think it's somewhat lazy to not talk about what they are trying to guard against, when we are supposedly giving the absolute maximum benefit of doubt.
Are we just concluding "their concerns were never real"? Because that probably runs counter the things that they have been observing and concluding.
If you believe Anthropic believes what they say they do, all of it makes sense.
Generally, in the past when tech companies have made outlandish claims that were not backed by evidence, they're later found out to have lied. This is an ancient pattern going back to the dotcom era and before, but for recent examples you need only look back a few years to the web3 era. If they're not lying, they can show it by producing the results they claim. Until then, they're probably just lying.
> If they're not lying, they can show it by producing the results they claim. Until then, they're probably just lying
Brilliant framework: Anyone making claims about the future is not just speculating, not just wrong, but they are lying.
Claim: GPT-4 is a PhD level expert.
Claim: LLMs "reason"
Claim: LLMs "understand"
Claim: AI will automate jobs away
None of these are "earnest" claims for a company to make (whatever that means). They're vacuous bullshit, and false. It's a bunch of wordsmithing papering over results that are either hidden from view or just unimpressive. Even if the person saying that shit believes it, that's not super relevant--if a company is advertising these capabilities then... just... doesn't do it, that's lying. Corporate statements aren't unilaterally determined by fallible individuals, they're reviewed, crafted products. They can be fairly critiqued as such.
> Brilliant framework: Anyone making claims about the future is not just speculating, not just wrong, but they are lying.
Not just anyone, companies in particular. If a company tells you it's building something to replace jobs, and then it doesn't do it, that company lied.
My company itself (possible only with AI) does the work of at least several dozen people across my hundred customers or so. Those jobs are now automated away.
Does that mean you're lying, or just overly confident (and wrong) in your speculations?
FWIW, I wouldn't put Sam Altman in the category of "earnest." I'm not sure if you just aren't aware that Anthropic and OpenAI are different companies, or if you're arguing dishonestly by trying to put sama quotes in here? But weird move in either case!
I'm glad you're seeing some success with this. I don't pretend to get it, maybe I'm too old and worn out.
I'm not sure there's anything "to get." But given your level of curiosity it's not surprising.
Web3 was absolute horseshit (and always was), so if you're blending AI with that based on the similarly grandiose claims and the extremely annoying boosterism, I think you should try hitting reset and engaging with LLMs from a cleaner slate.
EDIT: And, if it wasn't abundantly clear, fuck you and the goddamn fucking horse you rode in on, you horrible sack of shit. In the best faith, of course. Die in a Fucking Fire.
IMO they are using the cult messaging to distract the public so they take out all the oxygen in the room regarding people that care about the immediate impacts (climate exacerbation, ease of scamming, degrading job prospects, increasing income inequality).
Whenever real concerns are brought up against these companies they are always ignored while claiming the real concern is the fantasy of a machine god turning into skynet.
If they believe they're creating "a machine god" and that it's better it's their machine god than someone else's (which, given the other contenders, I tend to agree with), then all the corollaries you mention are mostly irrelevant.
Whether you believe they're creating a machine god is irrelevant. They believe that they are. It would be helpful if you could create an actually good argument for why they cannot or are not creating a machine god, but it turns out there are no good arguments for why it's impossible to do so. And so... they shall try.
Good to know.
Arms races always work out great for arms dealers. Less so for the average Joe.
Companies don't have to do that. If they're getting into actually dangerous territory, they can stop as soon as they want to.
If you don't believe that, or you don't believe that the frontier labs believe that, then sure, it makes no sense. But they probably do. The people at these companies literally dedicated their lives to building this specific thing that, up until people had to make tradeoffs between "that looks risky" and "that looks useful", virtually everyone agreed would be a dangerous technology.
What apparently many people on HN failed to appreciate is that the thing that makes it dangerous is the fact that it grows in utility.
It's like a prisoner's dilemma where one party is loudly lecturing the other about the obvious benefits of cooperation while also obviously working on defecting. They want to have their cake and eat it too. Maybe they really are the pure-of-heart Chosen Ones destined to lead us around the great filter, but I don't see why I should believe that's the case when their behavior is just as easily explained as maneuvering toward being the winner who takes it all.
Yes, this dynamic is exactly the one that anyone who's concerned about AI is concerned about. I don't know why you state this as if it's evidence against the concerns lol. Someone being concerned about the incentives of a situation doesn't de facto make them immune to those incentives, obviously.
The implication that someone who's concerned about an arms race dynamic could simply opt out of the system that produces that dynamic is simply confused about what arms race dynamics are. The entire point is that it's a trap, and it's a trap even if you know it's a trap, and even if you don't like that it's a trap. There's nothing dishonest or hypocritical about being in the trap: it is literally a trap –– that is what it does and why it is bad!
I'm confused by these comments that imply people believe Dario et al are "pure-of-heart Chosen Ones destined to lead us around the great filter." Who? I've never seen it. And any AI-doomer is probably of the opinion that the entire question of Dario's or anyone else's personal moral character is 99% irrelevant. Because, again, it's a trap. The dynamics at play are so much larger than whether someone irks people for their lecturing tone. I would much rather give my money to Dario, who seems like a generally good person, versus Sama, who seems like a complete snake, but I'm under no illusions that doing so changes the fundamental dynamics that are steering us to AI doom. I doubt anyone does.
And yes, obviously they are angling toward being the winner who takes it all. That is literally the trap. If you believe what they believe, yelling "let's cooperate!" while hurdling towards the finish line and tripping your competitors is the only reasonable thing to do. That is the problem.
I think you're reading some subtext into my comment that I didn't intend. Knowing myself, I assume the scare quotes there are just a bit of casual irony re: the insanely high stakes here. The word "concerns" as used by previous commenters doesn't seem equal to the context.
> The implication that someone who's concerned about an arms race dynamic could simply opt out of the system that produces that dynamic is simply confused about what arms race dynamics are.
You can, in fact, opt out. You can opt out and do your damndest to stop what's happening, throw every cent you have at it, bend any ear that will listen, make use of the fact that your voice (as Anthropic leadership) has some meaningful weight.
If you really believe that we are heading down a path that's likely to end poorly for most or all of humanity, and you are the kind of person who's inclined to favor saving billions of lives over saving your own skin when the stakes are still relatively distant, abstract, and generally unclear, opting out is obviously on the table as a grand gesture that burns your position in the race to show just how fucking serious you are. The sense of inevitability your comment shares with many others does not seem well founded---we have, for instance, not had a global nuclear war yet. Leaders in the 20th and 21st centuries have shown remarkable restraint.
If today's political and tech leaders are unable to think beyond this inevitability, for whatever reason, the worst outcomes essentially become a self-fulfilling prophecy to the extent that reality bears them out.
---
But yes, these people are acting the way they are for obvious reasons, obviously. My previous comment is reacting to the general disagreement over whether Anthropic actually believes what they say about safety, etc., or whether it's a marketing gimmick. The purpose of my comment is to explain that "it's hard not to be cynical" about actions taken by very rich and powerful people that are claimed to be in everybody's best interests but are indistinguishable from the actions they would take to maximize their future power and wealth. I think everyone ought to agree with that statement. It's not a value judgment; it's simply an observation of how it feels to be on a plane whose pilot appears to be robbing the passengers (including you) at gunpoint and is conspicuously wearing the only parachute on board.
I'll mention again the nuclear analogy. It is, believe it or not, possible for great powers, and even adversary great powers, to agree to limit the development and proliferation of dangerous technologies.
> The main secret is out of the bag.
This is not something you can do in a shed with a handful of GPUs just because you know "the main secret". To build something like Mythos you need tens of billions of dollars, massive amounts of power, enormous buildings filled with specialized bleeding edge computer chips that are made by (optimistically) a handful of companies with deep government ties. You need free access to all the intellectual property that humans have created and posted openly on the internet. You need all of this at each step, and to take each next step you (or somebody) needs to have taken the previous one.
For now, there are a million ways for a government to pump the brakes on this cycle.
You were literally just criticizing Anthropic as disingenuous for begging for this. Or is your position that people other than those near the front of the race can agree to limit development? And if so: provide evidence.
Note also a key ingredient that makes nuclear non-proliferation possible is that they're pretty much useless weapons. There is no smaller order nuke that's dramatically more useful than a large conventional weapon. That's not true of AI models, which appear to be monotonically useful as they become more powerful.
What is the difference? It's easier to make money with the AI you get at each incremental step toward potentially destroying human civilization (though, of course, it's debatable whether these companies really are making money as such).
So what? You are implicitly arguing that human civilization will be unable to resist engaging in a large-scale, coordinated effort to destroy itself, just to make a few bucks along the way. Is this true? I don't know. The point of the nuclear analogy is that we have previously shown that we can, under certain conditions, put the eschaton back on the shelf for some period of time, despite very real pressure to take more incremental steps toward doom. "But AI can write code" is not a refutation of the possibility that we could take a more measured approach to AI development.
There may or may not be such a point with AI: A point at which ever smarter machines provides no marginal benefit to security. If that happens, I do expect agreements, should any one of us still exist.
There are billions of people who have opted out of playing the game. Has the game stopped? Has any game stopped because the people not playing it decided that it ought to? Only with government intervention, which is exactly what you just criticized Anthropic for being disingenuous for requesting.
Is your position that they should just be smart bloggers asking for regulation, instead of the preeminent lab asking for regulation, and that would be either more ethical or more effective? If it's less effective, isn't it de facto less ethical?
What say you about the thousands of smart bloggers asking for regulation who are ignored every single day and have no tools besides their blogs to steer the outcome?
> burn your position in the race to show just how fucking serious you are.
This is incredibly naive. Literally no one who is unconvinced of AI doom would be convinced by this... because they already don't believe the premise. Such a gesture would be readily explained away as "you were losing the race," or "you got rich enough already." This is the attitude when any individual opts out of participation (see: Hinton) and it's ridiculous to assume it'd be different if an entire company did it.
Not to mention, that an entire company can't do it. These companies have boards of directors. They are accountable to shareholders. A CEO who wanted to do this would simply be fired and the company would carry on. This is one of the key components of the trap. Large companies are not under the control of people but of incentives. They are literally deliberately designed not to be under the control of individuals –– to be immune to exactly the type of behavior you think is possible.
And yes, nuclear weapons are analogous to AI in the arms race dynamic to create and proliferate them. They are probably not analogous in there exists a stable equilibrium in nuclear weapons due to "accidents" of their nature. There need not be a similar equilibrium among competitive AIs.
----
And yes, your comment lands in exactly the category I mentioned. You do not believe the AI doom fears, so the behavior looks one way. I do believe in the AI doom concerns, so the behavior looks another way. This applies equally to yesterday's actions, today's actions, and tomorrow's, including some hypothetical honorable self-immolation to slow progress: I would see that as concordant with AI doom concerns, you would not. You would find it "hard not to be cynical" about the fact they already earned a ton of money, maybe was losing a bit of ground in the race, so on and so forth. This is plainly obvious to anyone who has had to converse with no-doomers, who can only analyze other people's behaviors under their own belief system, so it won't happen.
The only variable of disagreement is around AI doom.
> The only variable of disagreement is around AI doom.
The source of our disagreement seems to be your belief that somebody can either a) believe "AI doom" is inevitable, or b) not believe it's possible. This is an obvious false dichotomy that's stunting your ability to engage effectively with what I've written, and also stunting your ability to understand the broader landscape of the issue. You are projecting this dichotomy onto everybody involved and understanding their behavior in that way, which is leading you to make other reductive and honestly bizarre claims—like, for instance, the idea that a sudden change of course from Dario Amodei at this moment in time would be broadly perceived as somebody who was already losing the race cashing out his chips. If you really do believe that, I have to assume it's because you're modeling your hypothetical observers as falling into one of your two extreme mindsets and assuming Dario, being a smart guy who knows a lot, thinks the same way. I believe it is—yes, I'm ready for the dopamine hit—naive to assume all people or even most people fall into one of these two camps. Naive, at best. Yours is a self-limiting framework for thinking about this stuff.
I encourage you to broaden your thinking and engage in less projection and ad hominem stuff in discussions like this. I probably won't reply to whatever you post next unless you can do a better job writing a substantive reply to what I've written here.
I actually already know how you'd perceive Dario's opting out of the race because I already know how you perceive Dario's requests for regulation, which is the milder version of the same logic, and is vulnerable to the same cynical allegations of self-serving, which you've already expressed.
Because from the outside, their behavior looks like a situation of "What if Microsoft/Apple put controls in place to make it impossible to develop an operating system using their OS?"
Unlike nuclear weapons, advancing in this arms race requires actually deploying the product over and over again. Deploying the product makes your advancements visible to your competitors.
It makes complete sense to try to limit the degree to which that's true.
The nuclear 'race' was based on the premise that the winner could use it to destroy all other racers (a faulty assumption, see the USSR among others). I will charitably assume Anthropic does not intend to literally destroy anyone and merely wants to become an AGI monopoly. But if AGI is so powerful, any monopoly would not be stable since the incentives for entry into the market are massive. Why would China stop developing AGI just because Anthropic has it?
or is it more similar to the Cold War, where there were obviously competitors engaged in the race?
And yes, agreed the equilibrium dynamics for AGI are very different (and far harder to predict) than nukes. That sounds like a good reason to be sure we get there first since presumably any potential advantage wouldn't go to the second or third runner-ups
"Ability to literally destroy the other entity" is not a necessary or even typical feature of arms races.
It seems that the frontier labs believe they're participants in a winner-take-all market. Therefore they're in "an arms race."
Winner-take-all markets do not require that the winner literally destroys the losers, but only that the winner enjoys disproportionate returns compared to their actual superiority.
Whether or not this is actually true is TBD, but I think you're naive to think the frontier labs do not believe this to be true.
As far as naivete, wouldn't it be more naive to take their EA claims at face value, rather than the more realistic assumption that they like money?
You're pretty explicitly saying that dominating the competition is not the type of "destruction" necessary to qualify as an arms race.
> As far as naivete, wouldn't it be more naive to take their EA claims at face value, rather than the more realistic assumption that they like money?
Huh? Greed is – quite obviously – the major driving force behind the arms race. That is not a mitigation whatsoever.
> Whether or not this is actually true is TBD, but I think you're naive to think the frontier labs do not believe this to be true.
P.S.: On reflection, it's even worse than that, because it'd trigger based on anything the user types or reads on any site. Someone mentions a "critical rendering path" and now you can't participate on that thread in the Blender forums.
Let's just assume it was "only" that?
It's unreasonable to assume they are aiming to upset people who are just giving them money in the way they want. It makes no business sense, for any company. So that has to be a byproduct.
Model training is one of the more expensive undertakings in the world right now and distilling models from competitors against the TOS is apparently something that is going on for very little money. Why would they not "just" try to take measures against that?
All they had to do was have a simple, transparent output "Sorry, that request is against our terms of service. This session has been terminated"
The vast majority of frontier research is about how to build better models, not about alignment.
All this longtermism though is harmful. There are real problems of data theft, bias, labor displacement, and environmental costs that are happening right now but every push for regulation and regulatory capture, and all the safety talk, is always focused on some speculative future machine god to distract from the current problems.
I'd have a higher opinion of these labs if the issues they openly talked about and worked toward where the real issues we face currently, not speculative defenses against some future AGI that may never happen in my lifetime. I'm less worried about "our new model might kill all humans in the future" and more worried about how we are going to address anti-competitive behavior, copyright protections, labor rights, and the energy impact.
Honestly, that respect for 'copyright protections' has somehow become a leftist shibboleth is bizarre to me and indicative that something has become deeply warped in our discussions around this topic.
Frankly, this appeal comes across as the same kind of impassioned plea that a missionary might make when begging the faithless to repent and come to Christ before it's too late. This weird religiosity some people around here use to talk about AI, ASI and AGI is bizarre. Take what I've quoted and replace the words "progress" and "ASI" with "sinning" and "the Book of Revelations", and the zeal becomes apparent.
Outside of that though, there are other issues right now that need addressed before we speculate about what might be possible with ASI in the future. If the potential for a harmful ASI is truly that near, and that great, then why push forward at all? Where's the push for a global stop order on development of this technology until regulation can catch up?
The talk of a potential future serves as a distraction from the very real problems people are facing in their lives today.
While Dario and team are worrying about ASI, real people are worrying about how they are going to continue to feed their family after wide spread layoffs set a very large portion of the population back into a lower quality lifestyle. Real people are concerned about water usage is draught stricken areas, the massive energy demand driving grid instability in their communities, or that the environmental and economic externalities of model training is being socialized while the profits continue to be strictly private.
What about the mass proliferation of misinformation at scale having a real effect on our democratic process?
Forgive me if I'd like to see those addressed first, and fast, before we start worrying about an unpromised future technology.
That being said, I can't help but experience a bit of Deja Vu over arguments like those around biorisk. I've seen the same exact things said in the early 2000s over widespread access to broadband and Google. When the anarchist cookbook spread around online and everyone was super paranoid about democratized terrorism, and we had big regulatory pushes for ISP level censorship and user tracking. Telecoms frequently argued that only they can keep the web safe, with strict and expensive regulations that naturally only those large heavily capitalized companies can afford to go through. Like the early internet and search, its just another way to lower the latency required for a human to find already existing public data
Well, very little of that played out. Turns out the math, for now, is the same, and information retrieval doesn't directly correlate to democratized weaponization. In 2001, a bad actor still needed a physical lab, precursor chemicals, etc to build a physical threat. Those same exact physical constraints exist today. The software cannot yet cross the digital-to-physical divide.
Keep an eye on the risk, by all means, but I don't see it yet as justification to cement a monopoly or oligopoly, nor do I see it as a reason to prioritize a risk of information availability over the climate and environmental risks that are far more likely to end the species.
If you have a sizeable bucket of money, it's so, so easy to get folks so distracted by (or invested in) movie plot threats that they totally fail to (or have a "plausible" excuse to fail to) notice the actual, lasting harm that you're doing to society at scales both small and large.
If Anthropic had pushed hard and nonstop since their founding to ensure that all LLM companies in the world were legally bound to stop all LLM development the minute any one of them called for a halt to work, then I'd give their claims about safety some credit. They've been screaming about "safety" and "alignment" for years, but -because LLMs are impossible to secure against code injection- their products are fundamentally unsafe and always have been... I just don't trust their claims about a commitment to actual safety.
My read on their recent calls for a global "stop work" emergency cord is that they're very soon to (if they haven't already) reach a point where they will not be able to produce products that are sufficiently improved over the previous versions to justify the level of investment required for their development.
My prediction is that Anthropic and OpenAI will get serious barriers to entry of new competitors enshrined in Federal law, they will call for a "pause" or a "slowdown" in new research for "safety" reasons, and the US will attempt to engage in economic warfare with any countries that don't agree to force their domestic LLM companies to stop working on those LLMs.
unless the bitter pill is gone, extraordinarily not this. The capabilities will be limited by the training data we can create to pull information and patterns from
and then we will still be limited by compute, space, and power
mass devaluing of labour isnt particularly believable when everyones predicting that all the big labs are gonna go under trying to subsidize tokens.
power consumption and global climate change should
ASI should be in the top 10k concerns maybe, but way below what to eat for dinner.
much higher on the fears is some hype guy pretending he has made this thing, and giving it access to too much stuff, which it then randomly deletes or misuses
it should also be in thes same range as "what if the dinosaurs came back and ate everyone"
theres tons of progress on that too. same with finding aliens
there are real present concerns to worry about, like genocides, concentration camps for immigrants, food costs next winter, ongoing wars in the middle east and europe, etc
all kinds of actually pressing stuff, that doesnt first require burning a couple trillion dollars and forcing poor people to pay through the teeth for their electricity
Their concerns are probably real but I don't think they're being totally transparent about their concerns. They don't want to be subject to regulation (until they have captured the regulator) -- same as every behemoth.
But there are also people who just oppose utilitarianism, like G.E.M. Anscombe. For instance, in https://integrityproject.org/wp-content/uploads/2015/07/mr_t..., she seems to grant that dropping the nuclear bombs on Japan was probably good from a utilitarian perspective (because it saved lives overall) and also to grant that bombing campaigns that necessarily entail massive civilian deaths (including, apparently, area bombing German cities) are morally permissible but still to argue that dropping the nuclear bombs was impermissible because it constituted murder ("intentionally" killing the innocent). But this kind of distinction, which I think is what actual anti-utilitarianism must come to, is hard to even consistently maintain, and I suppose many HN readers would find the effort quixotic.
It is relatively easy to take the proceeds of a massive fraud, buy a relatively small (as a percentage of the fraud) $ amount of mosquito nets, and save more lives than the lives impacted by your massive theft. Is this a correct application of the utilitarian calculus? What sort of data would we need a priori to do this calculation "correctly"? Do you think he had a careful estimate of the suicide rate of victims of ponzi schemes before perpetuating the fraud, or would any suicide rate have made the decision net [pun intended] moral, as any such victim of fraud would lead to >> 1 net purchased (so you would almost always net save lives).
The above is of course snarky. It is also a best-effort way of analyzing a notable utilitarian's actions. I do not think it would be difficult at all to use this type of argument to argue that SBF's actions net raised utility in the world. If only we all would become fraudsters, then we could truly live in Omelas --- a notable utilitarian paradise.
Now if we look at EA, the basic tenet of EA seems obvious -- basically just utilitarianism. And from what I've seen, in practice also, EA is a pretty big tent. I don't know the specifics of SBF's case, but I think essentially no one thinks that he acted correctly. I don't know how many mosquito nets he bought, but I agree that if he bought enough, it might be that he net raised utility, and if that is so, it's something to be thankful for. But it doesn't make him some kind of utilitarian saint unless he couldn't have done even more good by some other course of action that wouldn't have hurt the ponzi scheme victims and brought opprobrium on the whole EA movement
I think this being a reasonable utilitarian point to make is not a point in utilitarianism’s favor.
I don't think people are objecting to the EA idea that some charities are more evidence based than others so much as the distinctly EA idea that it would be more effective still to donate to charities like OpenAI
now its utilitarianism taken to the extreme. if you believe a skynet scenario killing everyone on earth is plausible then the "logical" thing to do is allow literally anything in the name of stopping it. that includes mass murder and dictatorship. the only thing that can balance the infinite negative value from an evil machine god is the infinite positive value from a good machine god.
thats the main difference today, one faction around sam and dario believes in creating the good ASI first and sacrificing all the world resources to do it before someone makes the bad one, the more pessimistic like yud want to stop all ai development to reduce the risk that an evil god is made to zero.
at this point its basically a religion.
isnt that literally his thing since the 90s or something?
I understand how one may wonder if there was a way to do that, but it feels insane to me that one would actually conclude that "yes, it is possible". We have examples everywhere showing that it is generally impossible to define a metric that correctly represents the underlying concept we want to measure.
Said differently, I feel like Effective altruism fundamentally starts by saying "I don't believe in Goodhart's law". Which seems intellectually dishonest to me.
> Look. I’m the last person who’s going to deny that the road we’re on is littered with the skulls of the people who tried to do this before us. But we’ve noticed the skulls. We’ve looked at the creepy skull pyramids and thought “huh, better try to do the opposite of what those guys did”.
https://slatestarcodex.com/2017/04/07/yes-we-have-noticed-th...
"Look. I see that it doesn't work. I want it to work, so I will continue trying, even if it fundamentally cannot work. I am not interested in thinking about whether or not it can work. I am interested in showing to the world that I am well-intentioned and trying to do something, even if that something doesn't make sense".
The longer non-snappy explanation is that "I will arbitrarily set numbers on things and call it impartial" obviously doesn't match EA's self-conception, that lots of EA cause areas are speculative and don't focus on numbers, that EAs that do focus on numbers do a lot of work to make sure the numbers aren't arbitrary, that EAs as a general rule don't claim to be impartial, and that awareness of Goodhart's law doesn't mean "never trying to objectively measure anything at all".
> I am interested in showing to the world that I am well-intentioned and trying to do something, even if that something doesn't make sense".
This is the kind of pre-conception that's essentially immune to reality. I hear the same thing about vegans (oh they say they care about animal suffering, but everybody knows about factory farms, they just want to feel superior to everybody else) or environmentalists (they say that climate change is a threat to humanity but really they just want to lecture us about our cars).
All I can say is that it doesn't match my experience, and that the effective altruists I've met spend quite a lot of time "thinking about whether or not it can work" and trying to learn from other people's mistakes.
> that awareness of Goodhart's law doesn't mean "never trying to objectively measure anything at all".
Goodhart's law doesn't say "never try to measure anything at all". It says "if you try to optimise for the metric, then your metric is doomed". What EA does is pretty much say "let's devise a metric and optimise for it". It does NOT say "let's measure something without influencing it at all". That is totally different.
Wikipedia says (happy to read your corrections if you think it is incorrect):
> Effective altruism (EA) is a [...] movement that advocates impartially calculating benefits and prioritizing causes to provide the greatest good. It is motivated by "using evidence and reason to figure out how to benefit others as much as possible, and taking action on that basis".
While I appreciate the idea of "trying to provide the greatest good" (difficult to go against that :-), my criticism is about the method.
* It is not very hard to convince oneself that if we stopped eating animals, then we would stop abusing chickens (did you know that tens of millions of chickens die during transport in trucks every year in England?) and emptying the oceans, and it would be objectively better in terms of animal suffering and for the biodiversity.
* It is not very hard to convince oneself that our CO2 emissions are literally going to get most of us killed, and that it would be globally better for us "humans who are currently alive" to do something about it. But there already, it's not entirely clear to me if the better outcome for life on Earth is to save the human species. Kind reminder that the human species is currently, measurably destroying all other species at a speed orders of magnitude faster than the extinction of the dinosaurs.
Effective altruism wants to do "the greatest good", but what is "good"? It may be good for a subset of humans to bomb another country and steal their oil, but obviously that would not be good for the subset of humans in the bombed country. It may be good for humans to find a clean magical energy, but that wouldn't change the current mass extinction for the other species (kind reminder that the current mass extinction has nothing to do with climate change, it is all about... well humans having easy access to energy and doing what humans do when they have cheap energy).
I feel like effective altruism says: "We can't define what the greatest good is, but we want to believe that anything is better than nothing. So we define a metric that we call 'impartial' (but that obviously isn't) and optimise for it, knowing that optimising for a metric defeats the purpose of that metric". Really it's rich people who want to do something good but don't want to bother getting informed and convincing themselves about what they want to do. "I'll give a ton of money and in return I get philanthropy points to share with my rich friends, but I don't want to have to think about what is being done with that money".
When someone invests a ton of money and energy into something they genuinely care about, they don't call themselves effective altruists, do they? They are just working for that cause. Effective altruism seems to be about rich people delegating the work of "doing something good" by donating some extra money, while they keep doing what made them rich in the first place (which almost always is something that is going against whatever I would consider the greatest good).
The people I have met at effective altruist conferences are not rich, though they lean upper-middle class. I've seen way more "enthusiastic broke student" types than millionaires.
> When someone invests a ton of money and energy into something they genuinely care about, they don't call themselves effective altruists, do they?
Well N=1, but I do.
(And also I've met tons of EA people who were not shy about investing all their energy in a cause they care about, even when all mainstream society tells them it's pointless.)
> “ ‘He is a prodigy,’ he said at last. ‘He is an emissary of pity and science and progress, and devil knows what else. We want,’ he began to declaim suddenly, ‘for the guidance of the cause entrusted to us by Europe, so to speak, higher intelligence, wide sympathies, a singleness of purpose.’ . . .You are of the new gang - the gang of virtue. ”
The real underlying motivation is that you can more easily get away with shady business practices if you cloak them in the language of great moral works selflessly undertaken for the benefit of mankind. Historical evidence tends to show the opposite outcome, but still, new generations unfamiliar with history will repeat this stuff with starry-eyed enthusiasm.
> “There had been a lot of such rot let loose in print and talk just about that time, and the excellent woman, living right in the rush of all that humbug, got carried off her feet. She talked about ‘weaning those ignorant millions from their horrid ways,’ till, upon my word, she made me quite uncomfortable. I ventured to hint that the Company was run for profit.”
Now the horrid millions are users of LLMs who submit morally dubious prompts and who must be gently steered back into the path of correct thought by suitable backroom manipulation, rather than direct rejection of the request.
The workflow would be; User asks for a thing. If it's a good thing, entity does the thing. If it's a naively bad idea, entity explains why you don't want that. If it's an actually evilly intended request, entity wags it's metaphorical finger or could even smite the user.
The problem is that flow isn't desirable if your entity isn't entirely god-like. It can bad even your entity is in ways rather far seeing.
Anthropic: Evilness detected. User has been smited.
This is the same exact industry that gives you paid usage limits as a unit-less percentage bar then gaslights customers every time the algorithm running that percentage bar changes or they lobotomize an existing model with increased quantization to squeeze a few more dollars out of existing hardware.
"Failing cleanly" might make their moated hype-machine look bad pre-IPO, so they certainly aren't going to do that voluntarily.
Anthropic doesn't care. The goal right now is simply to avoid any and all bad PR on the way to the cashout IPO.
And paternalism will generate far less bad PR than somebody using AI on something that does real damage and makes headline news.
same with bad press about their model sucking after they said its even better than sliced bread - sliced bread that will destroy the world if buttered
In practise though, how is this truly that different from system prompts?
They are essentially just trying to re-inforce that the system prompt must be respected.
Skynet does not fail.
It conquers.
Fail cleanly. Anything else makes it too difficult to rely on.
edit: Giving the absolute maximum benefit of the doubt I understand that they see themselves as "stewards" for lack of a better word. But the EA thing is really leaking through, and paternalism isn't a good look.