It boggles my mind how anyone can think otherwise. Dangers of superintelligent or even non-intelligent AI are the long-term result of the dangers of AI being developed and misused for human ends.
It's the exact same argument behind why we should be trying to track asteroids, or why we should be trying to tackle climate change: the worst-case scenario is unlikely or in the future, but the path we're on has numerous dangers where suffering and loss of human life is virtually certain unless something is done.
What sort of AI catastrophe do you think would happen?
The fact that we don't know how close we are is itself dangerous. It's like doing gene editing on pathogens without a proper understanding of germ theory and biosafety. That's where we are with AI.
Some AI dangers are certainly legitimate - it's easy to foresee how an image recognition system might think all snowboarders are male; or a system trained on unfair sentences handed out to criminals would replicate that unfairness, adding a wrongful veneer of science and objectivity; or a self-driving car trained on data from a country with few mopeds and most pedestrians wearing denim might underperform in a country with many mopeds and few pedestrians wearing denim.
But other AI dangers sound more like the work of philosophers and science fiction authors. The moment people start predicting the end of humans needing to work, or talking about a future evil AI that punishes people who didn't help bring it into existence? That's pretty far down in my list of worries.
Your ability to read this sentence right now when we have never met and may not even be on the same continent was once in the domain of science fiction. Don't underestimate technological progress, and specifically, don't underestimate the surprising directions it could go.
Some fantastical AI predictions will happen, most probably will not, and some utterly terrifying ones no one foresaw will almost certainly happen. The unknown unknowns should worry you, and AI is full of them.
Sure, but where should that rank in my worries relative to 'designer babies' and 'rise of authoritarian states as economic powerhouses' and 'corporations that can commit crimes with impunity' and 'rising medical bills' and 'widening gap between rich and poor' and 'far right extremism' and 'water shortages' and 'economic crisis wipes out my savings' and 'cyber warfare targeting vital infrastructure' and 'rising obesity' and 'voter suppression' and the many other things a person could worry about?
You know what would be useful for cavemen to ponder? The safety of fire. Or you know, just staying alive because there are more dangerous things out there.
The current state of so-called "AI" is our fire. It's impressive and useful (and there are real dangers associated with it) but it has no bearing on intelligence, let alone superintelligence. It's more likely that a freak super-intelligent human will be born than that we accidentally produce a super-intelligent computer. We produce a lot of intelligent humans, and we've never produced a single intelligent computer.
As it stands, we don't have the understanding or the tools to do anything useful wrt safety from a super-intelligent AI. We do have the understanding and tools to do something useful about asteroids and climate change.
That's just wrong. This very paper we're discussing has AI safety factors built-in: the AI is not supposed to be able to lie, and yet it exhibited deceiving behaviours anyway. That falls squarely under AI safety, and is a pretty useful observation to inform future attempts.
> The current state of so-called "AI" is our fire. It's impressive and useful (and there are real dangers associated with it) but it has no bearing on intelligence, let alone superintelligence.
That's conjecture. We don't know what general intelligence really is, so you simply cannot gauge how close we are. For all you know we could be one simple tweak away from current architectures, and that should be terrifying.
> As it stands, we don't have the understanding or the tools to do anything useful wrt safety from a super-intelligent AI.
Even if that were true, which it's not, we certainly won't develop that understanding or those tools if we don't start researching them!
> We do have the understanding and tools to do something useful about asteroids and climate change.
Suppose you had a time machine and went back to the 19th century to explain the dangers of asteroids wiping out all of humanity. They didn't have the launch capability or the detection abilities we do now, but if they were sufficiently convinced of this real threat, does it not seem plausible that they could be motivated to more heavily fund research into telescopes and rockets? Couldn't we plausibly have reached space sooner and be even better prepared in the present to meet that threat than we are now?
Instead of a time machine, couldn't we just use our big brains to predict that something might actually be a serious problem in the future and that maybe we should devote a few smart people to thinking about these problems now and how to mitigate them?
This all just strikes me as incredibly obvious, and we do it in literally every other domain, but somehow say "AI" and basic logic goes out the window.
This kind of elasticity in language use is the sort of thing that gives AI safety a bad name. You can't take AI research at face value if it's using strange re-definitions of common words.
This is not a redefinition, the harm results from the standard usage of the tool. If the AI is being used to predict the possible future behaviour of adversarial countries, then you need the AI to be honest or lots of people could die. If the AI concludes that your adversary would be more friendly towards its programmed objectives, then it could conclude lying to the president is the optimal outcome.
This can show up in numerous other contexts. For instance, should a medical diagnostic AI be able to lie to you if lying to you will statistically improve your outcomes, say via the placebo effect? If so, should it also lie to the doctor managing your care to preserve that outcome, in case the doctor might slip and reveal the truth?
There are no actual safety issues with LLMs, nor will there be any in the foreseeable future because nobody is using them in any context where such issues may arise. Hence why you're forced to rely on absurd hypotheticals like doctors blindly relying on LLMs for diagnostics without checking anything or thinking about the outputs.
There are honesty/accuracy issues. There are not safety issues. The conflation of "safety" with other unrelated language topics like whether people feel offended, whether something is misinformation or not is a very specific quirk of a very specific subculture in the USA, it's not a widely recognized or accepted redefinition.
AI safety is not a near-term project, it's a long-term project. The what-ifs are exactly the class of problems that need solving. Like it or not, current and next generation LLMs and similar systems will be used in safety critical contexts, like predictive policing which is already a thing.
Edit: and China is already using these things widely to monitor their citizens, identify them in surveillance footage and more. I find the claim that nobody is using LLMs or other AI systems in some safety critical contexts today pretty implausible actually.
None of what is remotely plausible?
And there's lots of actual, real scientists who think AI risk is a real thing we should be concerned with. Both within the field (e.g. Stuart Russel) and outside of the field (e.g. Stephen Hawking.) Are all of these scientists also talking with zero basis in actual science?
The intelligence is what determines the shape of these networks so they can even learn a useful pattern. That’s still gonna be humans for the foreseeable future.
Current AI research largely isn't focused on general intelligence, but the possibility remains that it could still spontaneously emerge from it. We can't even quantify how likely that is because we don't understand intelligence, so whatever intuition you have about this likelihood it's completely meaningless and not based on any meaningful data. We're in uncharted waters.
For a while I ran a project that ran a (scientific) binary on every Google machine in production using all the idle cores on the machine to do protein design (it worked!). At the end of the project we seriously considered making the binary instead a learning system that had access to 'sensors' (IE, signals from Google production machines) and 'outputs' (the ability to change production parameters) and just let that run autonomously. I figured Google prod is one of the few computational/networking systems with enough inherent complexity and resources that spontaneously emergent general intelligence was not inconceivable. However, the project was deemed too risky by SRE, rightfully so. Not because of emergent intelligence but emergent disruption to serving ads.
You missed the terrifying aspect. It's not terrifying that general intelligence emerged from optimizing for reproductive fitness specifically, it's terrifying that general intelligence emerged while nature was stumbling around randomly in the dark trying to optimize for a complete unrelated property.
Guess what we're doing with machine learning?
General intelligence didn’t poof into existence it happened over half a billion years and even then it came from fungi which is itself an intelligence.
Why millions of years? Do you agree or disagree that humans can develop technology faster than nature could evolve it on its own? It took maybe 70 years for computers to go from not existing, to matching humans on visual recognition tasks.
As you just acknowledged, it took nature at least billions of years to evolve humans. Is not focused technological development obviously orders of magnitude faster at evolving intelligence than nature? Does it not then follow that general intelligence is a lot closer than an argument based on natural evolution might imply?
What exactly is not logical about this argument?
Personally I figure that once you have a complex enough system with feedback regulation and self-preservation, something like general intelligence occurs spontaneously through the normal mechanisms of evolution (probably more than once in different kingdoms) because it provides a heritable survival edge.
I agree that the simplest properties of neural networks are an emergent property of multicellular life.
We must do something.
This is something.
Therefore, we must do this.
2. AI safety research attempts to mitigate some AI dangers.
3. Therefore we must fund AI safety research.
Yup, checks out.
The priests say virgin sacrifices pacify the gods.
Therefore we must sacrifice a virgin on every new moon. While we are at it, we should also build a palatial mansion for the wise priests.
Yup, checks out.
An argument that depends on equating priests and scientists is an argument that fails.
Edit: for instance, here's a computer science professor and neurologist who published a serious book about all the dangers of AI, and calling for funding safety research:
https://www.amazon.com/Human-Compatible-Artificial-Intellige...
Furthermore, AI safety programs are hiring actual AI researchers and computer scientists to actually do the work, so are you claiming that these people don't sincerely believe in the work they're doing?
These two things are in conflict. We could ignore both asteroids and climate change and according to the best known science there'd be very little impact for vast timespans and possibly no impact ever (before humanity is ended by something else like war).
Yes, also for the climate. Look at the actual predictions and it's like a small reduction in GDP growth spread over a very long period of time, and that's assuming the predictions are actually correct when they have a long track record of being not so.
Really stuff like asteroids and climate is a good counter-argument to caring about AI risk. Intellectuals like to hypothesize world-ending cataclysms that only their far sighted expertise can prevent, but whenever these people's predictions get tested against something concrete they seem to invariably end up being wrong. Our society rewards catastrophising far too generously and penalizes being wrong far too little, especially for academics, NGOs etc. It makes people feel or seem smart in the moment, and they can punt the reputational damage from being wrong far into the future (and then pretend they never made those predictions at all or there were mitigating factors).
That's just incorrect. The Tunguska event was a nuclear-weapon scale asteroid. These are predicted to happen once every hundred years or so. If it happens over a populated city millions would die. If a person wrongly concludes this was a surprise nuclear attack, maybe everyone would die, to say nothing of the real risk that the asteroid itself could be big enough to wipe us all out.
There's a lot of uncertainty around climate change, but changing climate patterns will certainly change resource allocations (fresh water, arable land, etc.). This will lead to shortages in places that were once abundant, which could easily lead to wars in which millions die.
> Really stuff like asteroids and climate is a good counter-argument to caring about AI risk.
This. This boggles my mind. Long-tail risks exist, and burying your head in the sand and pretending they don't just places millions of lives at risk, and potentially the entire human race. You don't have to think these are top priorities, but to dismiss them as complete unimportant is frankly bonkers.
Which had very little impact on humanity because it exploded in the middle of the tundra.
"These are predicted to happen once every hundred years or so"
What is predicted exactly, by whom and how were these predictions validated against testable reality given the postulated rareness? If they're so common then why is it so hard to name the last 10? I think in reality these events are very rare and will almost always happen over the oceans, deserts, poles etc where not many people live.
"Long-tail risks exist, and burying your head in the sand and pretending they don't"
They exist and I am not pretending they don't. I am saying that this style of reasoning in which an extremely unlikely event is unfalsifiably and arbitrarily assigned near infinite downsides in order to justify spending time and resources on it, is problematic and as a society we are far too generous towards people who do this.
"I can't name any such events therefore they're not worth thinking about" is a ridiculous argument.
The most recent one happened less than 10 years ago and almost 1,500 people were injured, and we completely missed its approach:
https://en.wikipedia.org/wiki/Chelyabinsk_meteor
Luckily, it again happened near a depopulated zone, but the damage was still extensive.
> I think in reality these events are very rare and will almost always happen over the oceans, deserts, poles etc where not many people live.
Most car accidents will happen to bad drivers, so if you're a good driver you don't need to wear your seatbelt, amirite?
The fact that an easily preventable event typically happens without much damage is no consolation when that's not the case.
> They exist and I am not pretending they don't. I am saying that this style of reasoning in which an extremely unlikely event is unfalsifiably and arbitrarily assigned near infinite downsides
Your mistake is thinking the likelihoods assigned are arbitrary and unfalsifiable. By your logic, COVID-19 was unlikely as most outbreaks are small and isolated, and the likelihood of a global contagion so remote it's not worth thinking about. Therefore pandemic preparation is a waste of time and money. Now that 6 million people have died, that view doesn't look so rosy in hindsight.
The calculations have been done based on known data:
https://en.wikipedia.org/wiki/Impact_event#Frequency_and_ris...
(There is value in doing research and ethical analysis into AI/ML statistical algorithms to prevent hidden biases or accidental physical harm. People working in those areas are producing real benefits for the rest of us and I'm not criticizing them.)
Who are you talking about exactly? Who are these alleged grifters and who are they grifting?
> There is zero actual scientific evidence to support their claims
What claims are you talking about, exactly? Let's get specific.
Computers just made the enforcement easier and with less opportunities to break out of it - for example, a sympathetic public servant no longer has the power to make exceptions since "the computer won't allow it."
An example given was a state wanted to spend less on low income benefits. They setup an overly complex computer system that decides eligibility and requires participants in the program to fill out multi page forms 100% accurately on a regular basis or their benefits are suspended immediately. There was no way to appeal this decision, only to re-apply.
Employees of the state agency no longer had any insight into why a form was rejected, and could not speed up the process to resubmit it because the participant had long term medical issues and needed potentially lifesaving prescriptions filled. As a result, many people who would otherwise be eligible for benefits had them denied.
This lead to a marked decrease in spending on social services and was considered a win by the state.
Anyone going through this rigamarole, that's always my advice. Keep pressing, they'll eventually accept. It sucks, but you have to play the game.
That isn't to say that corrupt public servants don't exist, and that they don't look nearly identical if you were to take a brief glance of their actions without looking at their reasoning, but that doesn't mean they're the same people or causing the same end effect.
"Sympathetic" and "source of inequity" are subjective. Letting someone get extra points on their test because of their race is equitable to some, unequitable to others. Who decides?
Equity, by its very definition, is unequal. Every single thing done in the name of equity is unequal. On purpose. Because the "job" of equity is to right inequalities in the system.
In practice it creates more freedom, because users have no choice in their top authorities (I'd have to move to another state or country if I don't like the laws) but often have a lot of choice in bottom level authorities (if the guy at the DMV is mean I can go to a different location)
I'm not saying it's strictly good, but it's much more complex. IF you are someone who truly believes the top level policy is good and perfect and has no problems, then yes, any kind of bottom level control.
But if your are someone who is harmed by the top level policies, you now have an escape hatch by which a public servant can help you overcome this.
For example once in NY I tried to register a car I had bought out of state and one DMV worker told me I could not because the previous owner was named Johnathan Smith but had signed his name as John Smith. Ridiculous, but it's possible they were following the letter of the law and an automated system would have said the same.
I simply went to a different DMV where a different worker took a more favorable interpretation. If it had been a piece of software making this comparison I would have been screwed.
I owned a vehicle for 15 years and it broke down on me, as a result I dropped insurance on it as I wasn't sure if I wanted to keep it or not.
fast forward 3 months and it turned out to be a relatively cheap repair so I went to get it tagged (it had expired while uninsured) and the woman at the DMV absolutely hammered me on the paperwork I had to show her to the point that even the agent at the insurance company was confused about what she wanted.
I go to another location and lo-and-behold it's no longer a problem. Maybe she was in a bad mood, maybe she didn't like the way I looked. I have no idea, but the point you're making is absolutely relevant in day to day life.
And if I can pontificate slightly further, inequities exist in life, there is no way to completely remove them. Giving autonomy to low level people and squashing the ones that abuse that autonomy is absolutely better for people overall, even if it isn't perfect.
I'm making this number up, but I swear that 90% of the time when I come across a company that's absolutely floundering under the weight of its own poor decisions, it's because those decisions are enforced by software such that people CAN'T work around it.
What ends up happening is doing anything crosses 15 silo's and each of those silo's absolutely enforces it's authority such that no single person is able to bridge the gap between even two of them.
What makes it hard is that there are legitimate reasons why you don't want a single person being able to write code, push it to production, open firewalls, open access to databases, add ACL's, etc. Depending on the industry, the need for controls should exist, and because there's SOME legitimacy there it gets pushed across the threshold past reasonableness to absolute pain for everyone involved.
---
And if I may, that's where the most dangerous solutions lie. When there's a legitimate use of them but they can be abused.
Well, you can solve some things. Internet and zoom went some way to solving "how do we work in lockdowns?"
https://www.cureus.com/articles/107920-increased-incidence-o...
The real social problem we ought to be solving is stripping bureaucrats and politicians of their ability to impose lockdowns on us at all. And there is no technical solution for that.
That's not logical. There's no valid need for traffic, but I appreciate Google Maps traffic reports.
So yea, inequality is a social problem, but it is amplified by technical problems. We’ve seen this time and time again, the Internet at large being the most glaring example.
We shouldn’t ignore this amplification effect while we wait to solve the social problems. Furthermore, computers are often viewed as some unbiased decision machine, which makes the problems worse.
I'm not really sure exactly what you mean by "feed AI inputs like race and it will make racist decisions", as that's an absurdly general claim that doesn't even really say anything useful. If you mean the model reflects the underlying stats that correlate with race, class, gender, then sure, yes, I agree the predictor will do that. To say that it's racist is an entirely subjective claim.
THe point of insurance isn't to avoid social ills, but to statistically minimize the impact of risk on individuals (while making a tidy profit for the insurer).
I'm sure if black wall street wasn't razed to the ground by a racist white mob (among several other attempts to gain stability, wealth, etc. by black people being destroyed by racist white anger) then maybe the model wouldn't need to "reflect the underlying stats that correlate with race...". But those underlying stats, and those correlations, didn't just happen in a vacuum, hewn out of the aether like a magical and consequentless thing.
Because it's fucking unfair, and humans hate feeling like they've been wronged for unfair reasons they cannot control. How would you feel if your loan got denied, you asked why, and you found it it would have been approved if you had been born in April instead of June? Don't pretend you'd love it and smile. You'd rant about it and tell all your friends how bad that lender is.
If the lender rejects you because your debts are too high or your income too low or something, or the zip code of the home you're mortgaging, that makes more sense and feels more relevant and in your control, at least.
Yeah, it's nice from the point of view of the lender, but the cost is too high to society.
Society feels so strongly about this that (in the USA) we've even passed all kinds of laws about what pieces of information are absolutely banned from being used to determine who gets a job or who gets a home. If you are a landlord or hiring manager you have to be aware of them so you can at least pretend to comply.
Sometimes I wonder if AI is already here secretly. Half of my replies on hacker news seem to basically boil down to teaching what humans are actually like.
I mean, I pay more for car insurance because I am immutably male. I don't feel great about this, but assuming that males my age are statistically more likely to be in crashes this makes total sense.
Likewise, I don't feel great that even if I were really good at basketball I'd be way less likely to make the short list of an NBA recruiter, because statistically I'm not going to be as good as someone two feet taller. It may not feel great, but it still absolutely feels fair too.
If people born in April instead of June were actually statistically more likely to be in crashes, I don't see how this is any different or unfair.
Because it’s radically unfair to do that! You’re seriously suggesting that some people should be penalized for the day they were born?
Taking the point more seriously, it’s hard to believe that the risks for whatever you’re ensuring have populations of people each month that are so different that a random person from each has a risk associated with them that birth month, as a signal, isn’t likely to discriminate unfairly almost as much as it helps. Naively, it’s likely to unfairly penalize almost half the population. Maybe you’re prepared to accept that because, hey, it’s right over half the time, but tell that to the vast population you’re penalizing.
And some things just shouldn’t be included. Health insurance, for example: isn’t it unfair to include risk of cancer caused by genetic factors in premium calculations? I guess you could say that’s different because parents could control birth month, but, man, that just seems so uncaring to me.
In the race example, let’s say there is a discrepancy in the underlying stats now, but it’s caused by structural problems in society caused by racism in the past. By using that as a signal to discriminate now it makes it more unlikely that the problems will ever be resolved (example: Black people cant get loans as easily, so can’t start businesses as easily, so the wealth gap is reinforced). If you think that’s ethical I would suggest that you are considering statistics too much and people too little.
Genetic risk of cancer cannot be included in insurance decisions (excluding voluntary life insurance) because a law was passed (GINA) preventing it.
It's not the role of the loan industry to make it easier for black people who appear to be a higher statistical risk to get loans. That's the purpose of government- to make it illegal to include genetic history in life insurance models- provided that society believes in that principle.
Another way to say that - one that appears less uncaring to readers - would be “yes, including birth month or race in insurance risk calculations would be unfair+, so it should not be done. If necessary, the government should step in to prevent this”.
+ you may squabble with the use of ‘unfair’ here - obviously it is not ‘unfair’ in the stastical sense, but that’s not what the word generally means in spoken English. ‘Inequitable’ would be better, but for some reason ‘equity’ seems to be evolving to mean ‘at all costs, produces equality of outcome’ and so is a victim to the same problem in the opposite direction. I chose the more common word.
[edit: I used insurance here, sorry - the same would apply to loans I think]
I do agree that if you expect corporations to act morally, then it would be sensible for them to be more careful in understanding and correcting long-term social ills.
It is trivial to bake an irrelevant distinction into a naive ML driven process.
Even if we set ethics aside, this is still a terrible idea because credit risk analysis is highly regulated in most cases. There is enormous regulatory risk to using birth month in a credit risk model. Particularly because hen a jury asks "what's an alternative causal model that could explain the bank's incorporate of this data point into their models?" their answer is going to be "absolutely none".
Instead, they look at who was approved, and maybe whether they repaid their loan.
So if a human making loan decisions isn't keen on giving out loans in View Park-Windsor Hills, a model trained to produce human-level performance won't be keen on doing it either.
Historically, at least in the United States, loans were unavailable to PoC, especially black Americans. Districts in many American cities were "redlined"[1], that is to say certain districts of cities were deemed "unprofitable" for banks. Redlining was policy and was targeted towards discriminating against black Americans, who were often victims to predatory loans with unserviceable interest rates, which commonly resulted in defaults. The defaults caused worse scores for people in the neighborhood and created a positive feedback loop. Historical housing loan data (and insurance data) includes this data. People today are affected by this historical data and this absolutely must be taken into account by the developers of these models.
Because of the entangled nature of real data, dropping "race" as a feature for training wouldn't solve the problem. Factors like zip codes (think of the redlined districts) would also influence the outcome[2].
Creating models, whose output can impact people so profoundly, e.g. can Jane get health insurance, calls for more reflection than just "numbers correlate!".
[1]: https://www.smithsonianmag.com/history/how-federal-governmen...
[2]: https://www.educative.io/blog/racial-bias-machine-learning-a...
Political solutions are necessary, maybe something like subsidized loans for people from formerly redlined communities to purchase and restore homes, or start businesses. Urban planning projects, like increasing mixed-zoning, pedestrian traffic, and good public transportation would help keep money in the neighborhood. Then there is the question is how to deal with gentrification and increase quality of living in a community without displacing people from that community. It takes a team of experts from various fields, the community itself, and goal-oriented cooperation.
Holy cow, so literally a horoscope? Lol.
How often have you talked to someone at an institution about some egregious wrong, and they defend it as "that's our policy?"
Towards the beginning of the pandemic there was a lot of this sort of stuff, say: https://www.nature.com/articles/s42256-021-00338-7
I hesitate to say "safe space", but... what if a group of people wants to come together discuss AI safety? If they'd have to regurgitate all the arguments and assumptions for everyone who comes along they'd never get anything done. If you are really interested to know where they are coming from, you can read the introductory materials that already exist. If the 99.9% of the world is hostile towards discussing AI safety (of the superintelligence explosion kind, not the corporate moralitywashing kind) there is some value in a place which is hostile to not discussing it, so that at least those interested can actually discuss it.
Using ML for object detection, object tracking, or prediction on an L2-L5 driver assistant system? AI safety research sounds like a capability you'd really want.
Using ML for object detection, object tracking, or prediction on an industrial robot that is going to work alongside humans or could cost $$$ when it fails? AI safety research sounds like a capability you'd really want.
Using classifiers or any form of optimization for algorithmic trading? AI safety research sounds like a capability you'd really want.
Building decision support systems to optimize resource allocation (in an emergency, in a data center, in a portfolio, ...)? AI safety research sounds like a capability you'd really want.
Hell, want to use an LLM as part of a customer service chatbot? You probably don't want it to be hurling racial slurs at your customers. AI safety research sounds like a capability you'd really want.
Unfortunately, now "AI Safety" no longer means "building real world ML systems for real world problems with bounds on their behavior" and instead means... idk, something really stupid EA longtermism nonsense.
The whole point is that the technology itself and its capabilities are not well defined, people are constantly inventing new applications and new methods now at breakneck speed, so the question of how to mitigate its risks is going to be at least as squishy a concept as the underlying tech + applications.
Before 2016 or so, "AI Safety" meant the types of things I listed above: how to design AI systems that are safe to use in safety-critical settings.
Unfortunately, the term "AI Safety" has been overrun with effective altruist longtermism folks talking about something closer to https://en.wikipedia.org/wiki/Existential_risk_from_artifici... which is ill-defined and, frankly, usually completely vapid nonsense.
> so the question of how to mitigate its risks is going to be at least as squishy a concept as the underlying tech + applications.
No, no, no!!! AI safety methods -- methods that really increase confidence in real systems -- are not generic. ALL of the useful AI safety work I know of is deeply related to one or more of: the specific model architecture, the specific optimization algorithm, the specific data, or the specific application. Almost always all of the above.
Pressing the idea that AI is dangerous makes it seem like these companies are even more powerful than they are and could drive up their stock price. When the AI Safety people get into some conflict and get fired they are really doing their job because now it looks like big tech is in a conspiracy to cover up how dangerous their technology is.
Suffice it to say, ideologues desperately want control over everything AI/ML. That's the real danger.
Well that's an interesting way to misrepresent an entire important field of research based on what a few idiots said. There are serious people in that field who aren't addicted to posting on LessWrong.
Is that actually true, though? It's true that a higher fraction of the people in that community give credence to the intelligence explosion hypothesis than pretty much anywhere else. (This is what one would expect, since part of the purpose of LessWrong is to be a forum for discussions about super-intelligent AI.) But even if the intelligence explosion is a terrible, absolutely-wrong theory, that doesn't prevent the people who hold it from being open-minded and tolerant of independent thought. Willingness to consider new and different ideas is something the LessWrong community claims to value, so it would be a little bit weird if they were doing way worse than average at it. And AFAICT, it seems like they're doing fine. Some examples:
- Here [1] is a post critical of the intelligence explosion theory. It has 81 upvotes as of this writing, and the highest upvoted comment goes like: "thanks for writing this post, it makes a lot of good arguments. I agree with these things you wrote" (list of things) "here are some points where I disagree" (list of things). This may even be the original post you were talking about in your comment, except that it doesn't start with an apology.
- LW has 2 different kinds of voting: "Regular upvotes" provide an indication of the quality of a post or comment, and "agree/disagree votes" let people express how much they agree or disagree with a particular comment. Down-voting a high quality comment just because you disagree (instead of giving it a disagree-vote) would be against the culture on LW.
If you're already sure that LW is wrong about superintelligence, and you're trying to explain how they became wrong, then "those LW people were too open minded and fell for that intelligence explosion BS" makes more sense to me than anything about suppression of independent thought.
[1] https://www.lesswrong.com/posts/zB3ukZJqt3pQDw9jz/ai-will-ch...
But the A.I. hype is out of hand. "A.I. Safety" research is the worst of it, as it suggests this technology is so powerful that it's actually dangerous. The other day I was almost to write a comment on HN to a post from lesswrong where they apologized at the beginning of an article critical of the intelligence explosion hypothesis because short of Scientology or the LaRouche Youth Movement it is hard to find a place where independent thought is so unwelcome.
Let's hope "longtermism" and other A.I. hype goes the way of "Web3".