"Although the torment nexus may torment us in the short term, over the long term there will be fewer people to torment as we improve the capitalization of human's torment so that fewer people will be available to torment duing to the reduction in overall torment-forces globally, and thus, with less people being tormented, surely that will increase the over all torment reduction of the nexus. Win-win: less tormented people means less torment!

Or are you telling us in the hope someone makes you such an offer?

When someone isn't competent to make decisions for themselves it's ok to pressure them into engaging in behavior that will lead them to get to a condition where they can start to make decisions for themselves. In Silicon Valley they have let their guilt be leveraged by a corrupt government system that lets grifters steal billions of dollars per year in 'homelessness funding'. Those same grifters fund political groups and bribe politicians. Anyone that questions it is accused of wanting to murder homeless people.

There are many, many places in the world that are much poorer and don't have homeless encampments all over the place.

I'm not as positive as you are about what makes California (and Seattle) create/attract so many, but it's definitely an outlier compared to "everywhere".

Like, do you know many addicts or seriously mentally ill? Have you tried pressuring them into being better? How did it work out?

However, female and intersex/trans people are a very small percent of the homeless population. So while PTSD is statistically plausible within that sub-group, we are now taking a moderate percentage that would get PTSD from exposure to a potentially triggering event (say 15%), with a small percentage exposed to that event, among a very small percent of homeless people overall. Just giving this the highest numbers I can substantiate in your favor, it would produce 1.2% to 1.5% of the overall homeless population developing PTSD from triggers that are statistically due to homelessness PER YEAR (aka you can be raped if you have a home, but you are more likely to be raped if you do not have a home). This would accumulate to 4-6% out of about 25-50% prevalence of the PTSD cases found in homeless people. This is significant, so your theory is plausible given the available data.

However it would still mean that PTSD is 5-10 times more likely to cause homelessness, overwhelmingly due it's contribution to rates of hard drug addiction, than it is to be caused BY homelessness.

The weather is pretty nice, much more stable than in the east coast.

What you can do when it already happened is kill them or make them die in some ways, jail them or try to support and/or treat them.

I hope it goes without saying that letting them just die or killing or jailing them is not an option in a western society.

Prevention is much more favorable of course. For example, don't let people become homeless easily. But even then, a significant amount will become homeless.

On the street they have access to almost endless support, at least in Silicon Valley, but much of that support is contingent on things like not openly using heroin or not repeatedly getting so intoxicated that they defecate or vomit on themselves, so they refuse it.

I don't care how liberal you are, no society is functioning when it allows people to get on BART and pee on themselves and defecate on the cloth seats and walk up and down the aisles dripping feces and urine on the floor. That is allowed and nothing is done about it for fear of upsetting homeless 'rights' groups who are funded by the tax dollars meant to help homeless people.

1. Widen the definition of 'homeless' to cover people that nobody would consider homeless in every day language. Include anyone who is staying with a friend, living in corporate housing, staying with a family member, etc. Include anyone you possibly can that is unlikely to be a drug addict.

2. Narrow the definition of homeless. Don't consider people that almost everyone does consider homeless. Make sure to target the subgroups of homeless people that are very likely to use drugs.

3. Narrow the definition of drug use. Make it as hard as possible to count someone as having used drugs. Raise the evidentiary bar so high that even when you have evidence they are a drug addict it doesn't qualify them as a "drug addict" by the studies definition. Bury that definition so it's technically there but you have to go through the study carefully to find it.

Lets pick a study in a big important journal and check it for the above:

How about CASPEH: "Illicit Substance Use and Treatment Access Among Adults Experiencing Homelessness," JAMA, 2025

JAMA is a very prestigious journal:

Who counts as a substance user: Only meth, nonprescription opioids, cocaine/crack count as addictions. Cannabis, alcohol, benzos, hallucinogens, inhalants are excluded.

Someone who has a valid prescription to opiates, even if they are being abused, cannot be methodologically counted as a drug user. THIS INCLUDES METHADONE.

What counts as addict: "Regular" = 3+ times per week. A twice-weekly meth user is 'occasional use'. They might take a gram and stay up for 3 days twice a week, but they are 'occasional user' and don't count in the 37% addict number quoted by this paper.

What counts as addict: Last 6 months only. If you admit you used fentanyl every day for the last 10 years but claim you got clean 181 days ago, you are not a drug user according to this study. If they have been diverted from heroin onto methadone 181 days ago you do not count.

How did they check? They asked each person face to face whether they used drugs and which ones and how often and just took their word for it.

The real numbers from the raw data the study used:

41.6% of homeless self reported 2x per week or more hard drug use before they became homeless. an additional 23.2% claimed they started regular use after becoming homeless.

That's about 65% already from their own data, 2x what the headline pretends.

Excluding high use subgroups: to be included in the data, the homeless person had to prove they were giving informed consent. Then they additionally had to be willing and able to spend 45 minutes or so filling out the survey. The people with psychosis or who were intoxicated at the time of the interview were excluded. How many people have you walked past and they were incoherent, unconscious, or high? They were ignored in this study.

Homelessness is created by high rents. High rents are created by high local productivity. The people who are productive enough to make it over the rent-bar are housed, and their marginal productivity above the rent-bar serves to raise the rent-bar (talk to any landlord about how they set rent prices: it's the local wages, stupid). This then pushes a greater proportion of people below the rent-bar, and then they are homeless.

There are few places in the world more productive per square foot than Silicon Valley, ergo one must be extremely productive in order to remain housed there, ergo it has an extremely bad homelessness problem.

That's why people making $200k+ (an extremely productive person by national standards) need public housing assistance in Silicon Valley (https://www.reuters.com/article/world/palo-alto-considers-su...). Which of course actually just... raises the rent again...

The charities may be full of grift and whatnot, but you can remove them entirely and you will not solve the fundamental problem.

Wrong. High rents have many reasons and also good policies to get them down and have affordable housing for more people, also in otherwise expensive and popular cities.

When you combine that with spiking wealth in Silicon Valley, you’ve got the perfect storm in Silicon Valley. With OpenAI and Anthropic both based in SF, it’s going to be worse than ever once they IPO.

It was actually the post-war housing boom that was abnormal, fueled by aggressive federal funding of suburban buildout. US municipalities are now broadly doomed to bankruptcy trying to maintain sprawling infrastructure that was never cost-effective to create in the first place.

Look at Austin TX. Influx of new building, prices stabilized, and now construction is dropping off a cliff. No one can afford to build into a stable or potentially falling local market. Construction will start again when local productivity grows enough for people to bid the prices back up again.

Therefore, government policies making sure people with lower income can still get housing in a city they grew up in, where they work, live and have family, is important.

Yes, one part of this is also building new housing and making sure that housing is built efficiently in terms of cost (but not throwing them on the free market, but making sure a certain % of it is desginated as social housing).

Rents cannot (at meaningful scale) exceed local productivity, or else there will be widespread vacancies and landlords will reduce their rents.

Rents cannot (at meaningful scale) fall below local productivity, or else landlords will simply raise their rents.

Go talk to a landlord about how they site prices.

Step 1. Look up income of your target demographic

Step 2. Eat 30% of their income

Can you show me a place with high local productivity and low rent?

Can you show me a place with high rent and low local productivity (that's not e.g. a vacation destination for people with high productivity elsewhere)?

You can certainly artificially suppress prices with rent control. But that just gets fed back into the system in the form of reduced supply, so it ends up raising rents on the non-controlled housing stock anyway.

Put the following prompt into your favourite AI and enjoy the reading:

"How do rents relate to productivity in the area? How does Vienna compare to San Francisco?"

Try this one: "Create a simple analysis approximating how much higher rents would be in Vienna if not for the aggressive public subsidization of supply"

In any case, Vienna has a good system! But it's not that it avoids high productivity → high rent, it's that it splits rent into subsidy costs and rent.

In other words, the amount of subsidy + rent required will grow with local productivity.

Yes, it can be done, and if we want to have a reasonable future, it should be done.

Vienna's solution is fine but has escalating costs and isn't necessary. Land Value Tax is better.

I can’t. Immigration and language barriers alone would prevent (even though I have family there).

My point is just that Vienna doesn’t have to deal with tons of people constantly showing up.

Now make a law that makes construction cheaper. Or make a law that makes construction more expensive. What happens to rent and mortgages, even though productivity hasn't changed at all?

Productivity (or, rather, wealth) of the occupants is a very strong factor, but it is still just one of multiple factors.

No, you actually cannot do this in any lasting way.

The price of land would fall to the level required to make it construction viable again. Or the city would be abandoned, which is not generally the outcome since cities have so much shared capital investment that cannot be moved elsewhere.

Manipulating the real estate market via direct economic manipulation, collusion, or regulatory capture is so common that the entire concept of manipulating the market to earn higher profits is called https://en.wikipedia.org/wiki/Rent-seeking.

It's not just that it can, it's that it does so often it's taught in Econ 101. That you don't know this implies you have never taken an economics class or that you forgot what you were taught, and in the meantime certainly have not spent any real effort in learning about economic theory. I think you read a lot of Ayn Rand and right wing libertarian popular books about the free market and how it's perfect. Stop posting confident nonsense.

The claim is not that rents are always at their optimal or most efficient level - they're not! The claim is that you cannot (at meaningful scale) charge people who earn $1,000/month in income $2,000/month in rent to live in a place. They will have to move.

Did you take any arithmetic classes?

And any case, you should stop mind-reading. You're bad at it. I'm not a Randian nor a right-winger nor a libertarian nor free-market idealist.

Your mistake was thinking that I, the corrupt landlord who pays bribes, cares about a macro analysis of every all the properties, or that some analysis about the current distribution of incomes and possible rents will stop me from doing exactly what you say is impossible. I don't own every property in the city, especially the ones that aren't built because I capture zoning, so arguments about average GDP and average rent and whatever else do not stop me. Your analysis is useless.

This is the only feature that's relevant. If productivity goes up, you can charge more. If it does not, you cannot.

You may do some corruption or some zoning or some whatever else you want, but at the end of the day, if productivity goes up you can charge more and if it does not you cannot charge more.

Hope that clarifies!

Productivity is per-capita. Generally estimated via GDP per-capita.

At year 0 we have 500 people who want $2000 a month homes. In year 10 we have 750 people who want $2000 a month homes. Without changing the income distribution or economic output per person at all (and thus not changing productivity at all by definition) simple population increase will increase demand at every level, and now we have 250 additional people willing to pay $2000 but no increase in apartments formerly high enough quality to demand $2000, pushing $2000 rents down market. In the meantime there were 1000 people able to pay $1000 and now there are 1500 people able to pay $1000. They are likely to be forced out of their current neighborhood, so while they could only afford $1000 before, they will change their behavior and pack more people in by having multiple roommates or converting common spaces or dens into bedrooms. They still pay $1000 but the space formerly rented for $1000 may cost significantly more.

Population increase often happens as a result of more jobs being available. More jobs does not imply an increase in productivity.

Honestly you keep saying productivity, I think you mean total economic output normalized to housing available within a reasonable commute radius or something similar to that. In fact for the same economic output, lower productivity can cause higher rents because you can have more people employed who are forced to pay a higher fraction of their income for housing. Higher productivity at the same total economic output would cause houses to be abandoned as people have to leave to find work. This is less stark than it sounds because of service jobs, but it would still happen.

Like, a pure researcher for a corporation might have some genuine overlap and shared interests where breakthroughs in the research benefit both parties, but an ethicist?? For a legal construct that's defined by and bound to sociopathically pursue profit? Come on now.

At typical corporation, something like a Chief Ethics and Compliance Officer (CECO) would be largely focused on legal compliance - making sure company personnel aren't engaged in bribery, fraud, insider trading etc. The value of this role (and the medical example) is to prevent lawsuits and/or prosecution.

For a company like Meta, an ethicist is likely concerned with the inherent ethical concerns of running a social media company, which impacts the mental health of its users, many of whom are minors. For AI companies, the ethics of releasing models that are capable of doing dangerous things. In name, the job probably involves giving recommendations about how to ethically implement features in the platform to minimize harm. In practice, the job probably accomplishes very little.

I think a lot of it depends on the size of the company. Startup? Sure, maybe it's a real thing.

Gigantic company like a member of FAANG or MANGOS? absolutely a PR move.

I can't disagree with that, reminds me of the quote from the movie Rounders: If you've been at the table for 30 minutes and you don't know who the sucker is, you ARE the sucker.

ie: blaming her for anything instead of the decision makers is... really weird?

now they are exactly like every other company that has zero ethicists on staff and expect the employees to weigh the ethical ramifications of their work.

I’d argue a lot of them expect the opposite. To suppress the ethical concerns.

Even in recent memories you had Mark Zuckerberg telling his employees to use pirated sources. And executives at fossil fuel companies have to act unethically.

Do you believe that no one in management in these fields pushes the importance of ethics?

Food-testing: There's literally a parasite going around right now. Look at the response from some of the for profit businesses that was allowed to happen.

Hospitals: It is not ethical to deny patients care based on their insurance.

Clergy: The cover up of child sexual abuse scandals.

Childcare: lots of examples of sub-par care given to save money.

There isn't much ethical merit in building a wall around a city with guard towers, attack dogs, and anti-personnel mines to keep the citizens in so that they can contribute to your economic vision, for example.

All of that said, no idea what this person was doing. Probably involved a red stapler.

The capitalistic world state system was only completed in the last 50 or so years and was of course seconded by the Stalin machine. Its purpose is in the first instance to keep working populations in place, which is why all the states agree to it as they form. The system holds solid where capital moves (as did 'communism' and Comecon) but the peoples don't.

It is only when you think of individual states and not the system that you can fail to notice that we get the same result if they all try to keep their citizens in, and if they all try to keep the others' citizens out.

But that's just one extreme example. States based on Marxism-Leninism and Maoism in the 20th century often required things like internal passports to travel within a republic's borders. If the state didn't want you visiting family in another part of the country, you weren't visiting family. That still applies to some extent in a few modern single-party socialist states.

My larger point is, a lack of ethics consumes most economic systems, because you can't easily gauge ethics like you can quarterly profits or the output increase brought about by the central committee's five-year plan. Humans tend toward math over empathy in such situations.

It's performative. She got paid more to shill for a different massive amoral mega-corporation. Don't be stupid.

/s

So, pseudo-philosophical mumbo jumbo. There is of course no-one responsible for the ethical framework of the company's actions, because it is a company, and therefore amoral at best and immoral at worst.

But it appears that is not how your world works. By default any company owner is a homo-sapiens, but a machine that only wants money. Tragically, your cynicism is creating the very world you hate.

By contrast, how many conversations have you personally had with people working at a company believing a profitable action was firmly unethical, but felt forced to behave that way due to "business realities" or some other whitewashing?

On the second point: my experience is rather that there is quite a variance in what is consider moral vs immortal and that often drives different decisions (in the same place, some do certain things, other don't and certain experise might not be available for X therefore). I cannot say that I have come across many situations where people did something they considered firmly immoral but felt they needed to go along with it (trying to recall some). Yes, finding justifications happens, too, but that isn't necessarily whitewashing as people do disagree to surprising levels.

(Changed to moral vs immoral, because that was the starting point.)

It’s much easier to disregard the human cost of decisions when you are separated from said humans by thousands of miles & billions of dollars.

History shows this is not the case.

I'm happy for you to live in a place where the local economy is so vibrant. This is what capitalism should have been. Instead we got WallMart.

The problem is that corporations completely subvert any individual morality that might indicate that generating money is the only moral thing to do. Typically in small businesses. I comaider small businesses owners in general are a lot closer to labor than to capital owners in general terms, especially when their labor is still a large part of what keeps the business going.

Corporations are not that, those are hyper optimized for generating profits, all else be damned. Preferably if all else is damned.

> But it appears that is not how your world works. By default any company owner is a homo-sapiens, but a machine that only wants money. Tragically, your cynicism is creating the very world you hate.

They said companies are amoral at best.

When it comes to how humans interact with AI we can ask things like "would it be better if people could detect AI output?" "how can we alleviate skill loss from excessive AI reliance?" Things like this can decide how the models behave. Imagine if taking concerns like this seriously leads to models that are easy to learn from. At the moment people seem to be moving away from that so as to make distillation difficult, worsening this concern.

I'm not going to try to talk about machine consciousness, but these other things are definitely important questions where someone digging into them could make models interact better with society.

It seems like the pattern is almost always the same: the ethics team exists quietly in the background, and the only time anyone hears about them is when they either publicly leave the company or start a campaign against it.

So what exactly is the business case? Is it actually worth maintaining an ethics function just to be able to say you have one, when the only time it becomes visible is when it goes against the company publicly? And if this outcome is so predictable, why do companies keep hiring these people in the first place?

Ethics is also a containment vessel internally.

With an Ethics dept, concerned employees can rest easy knowing someone else is responsible, or if they raise concerns, they have someone to route and advocate their concerns, rather than adding noise to the chain of command. If the dept. ends up ineffective after some years, employees can blame the department, or have renewed hope when personnel turn over. So (slow) turnover is good from this perspective.

I'm having a hard time deciding what part of this is the most ridiculous. Contenders include:

- A political Science major lecturing on ethics.

- Meta and ethics in the same paragraph.

- A company who's CEO has credibly threatened (publicly) the livlihoods of hundreds of millions of people globally so much as paying lip service to ethics as a concept.

- Any news outlet feigning credulity hard enough to deadpan reporting on any of this.

Is it possible to roll one's eyes hard enough to injure oneself?

Someone challenge me.

https://en.wikipedia.org/wiki/Controversies_of_Nestl%C3%A9

However, with all things AI, this is a very weird situation because CISOs are let go when an organization is breached by external attackers; in this case internal attackers -- which were not human, let alone employees -- inadvertently breached other organizations. I don't think the normal conventions apply anymore.

It’s easy to theorize ad infinitum otherwise

She spent 6 years at Meta. I get the sense there are other factors at play than "ethics departments are just PR stunts". There's no way she isn't already aware of that. Unfortunately, the article is light on any other details.

It's possible she came to view her role at Meta as having enabled the company to continue its bad behaviour and decided against doing the same thing at OpenAI. Alternatively, things at OpenAI could be so bad that an ethicist who stuck with Meta for 6 years decided she couldn't stay any longer. Either way, from a PR perspective, this is spherically awful for OpenAI.

This comes at a time when there is huge pushback against all AI companies due to, in large part, ethical concerns. As with Flock, there is the threat that physical infrastructure, such as data-centres, could be attacked if nothing is done to win back public opinion. OpenAI would be wise to take Bakalar's departure seriously.

Again, look at Sam Altman and tell me with a straight face that he's hiring goody-two-shoes to fill these seats. He knows where the puck is headed, OpenAI's use by the state will be ugly and they need "ethicists" to help assuage the PR disaster. Considering that Meta's still afloat, she didn't completely abdicate her responsibility there. She might be just what OpenAI needs to avoid being lambasted for military and surveillance use.

I don't think ethics is fueling the outrage. It's people not wanting big noisy data centers close to their communities, driving up utility costs. Nobody would care if these things stood in Africa. Just like people don't want wind turbines close by but have no issue with them being abroad or in $remote_location or uranium mines polluting a far away country.

Also, you should read Careless People, if only because Facebook is trying its darnedest to censor the author.

As for OpenAI, I already hated the company pre-Sam Altman takeover because AI safety seemed like an excuse to brush away ethics concerns (e.g. "stealing training data is fine because if we don't withhold model weights the model might do something evil"). Post-Sam, I hate the company because they are now ignoring both the AI safety and ethics concerns.

...if you're wondering what the difference is, AI safety is roughly "making sure the AI doesn't do something bad" and AI ethics is "making sure we don't do something bad making the AI". There's a bit of a rivalry between the two, mainly because the former group managed to present their concerns in more business-palatable ways and thus won all the CxO level arguments. Even the most decelerationist AI safety takes are ultimately criti-hype: OpenAI's model breaking containment and hacking HuggingFace is an awful safety outcome, but it's also good marketing for an industry beleaguered by bubble concerns. You can't turn "AI is trained on stolen data" into hype.

Even in the target-rich environment of late-stage kleptocratic surveillance capitalism, it is hard to pick out a less ethical firm which has done more moral and direct harm, to more people, with as durable a record of defying all attempts and internal and external at harm reduction.

Never use their products. Never partner. Never participate.

He did an interview on Search Engine that I thought was interesting,[1] but it definitely seemed like there was more to the story than just, "Facebook needed to be ethical, and this guy was so good at ethics that he decided to help out."

[0] https://chochmat.org/rabbi-zvika-krieger/

[1] https://www.searchengine.show/what-does-it-feel-like-to-beli...

That's an interesting data point.

It takes a surprising amount of time for nepo babies to realize life is about more than making sure you're doing at least as good as other nepo babies you grew up with.

This is a very optimistic take. Most people don’t take this route, so without evidence I don’t think it should be a first guess.

No information does not mean “everything was good”. Usually the opposite.

the only ethics that actually matter come from legal.

https://xcancel.com/bradlightcap/status/2087211567012032862

It's hard to read anything into it.

My sense is that it is radically shifting from a fluffy marketing arm to a department expected to contribute meaningfully to development and justify its impact. I would expect an ethics team to build frameworks that can help train/eval the model that the company spend millions of dollars and months training is going to be aligned to the ethical stances the company chooses. If they can't, the waste to time and money is huge if the model requires retraining for ethics reasons. That's a different job than pondering roko's basilisk or whether AI is alive.

The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.

Funny thing, right around the beginning of the big AI takeoff, all the AI ethics people who visibly thought their job was something other than marketing got driven out of the big firms, and often the industry entirely, because they were in the way.

So, if too many think their job is marketing a few years later, well, there is a reason for that.

So it’s entirely possible you end up in a situation where you have a highly paid, fairly prominent head of ethics whose actual job is mostly just saying they’re the head of ethics, with little actual power or influence.

Ambitious people won’t tolerate that for long and they’ll move on

Haha, how refreshingly naive. What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from?

> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.

Or it's exactly those people who will be left. We will read about it in your book.

the CEO agreed because (I imagine) he really needed someone to take that position. and as everyone should probably have seen coming, the first time he really wanted to send out a release bugs and all, he called the guy into a room and pretty much browbeat him for an hour about how making the promosed release date was more important than making a good release, until he said "fine but I'm not responsible if it breaks".

the CEO held this up as an example of how he had kept his word not to send the release out without the guy agreeing to it. that startup, needless to say, is long dead.

This could be a lesson for us engineers to be a little less glib and a little more introspective, especially as you gain more power and influence in an org.

From the (ethics) team obviously, because the investors just have goals and aren't a team!

Big irony marker of course. But remarkably, in human history, this line of thinking would not be unheard of. Of course, I don't think it easily adapts to companies, so I don't really disagree with you.

Remember that MTV show offering people like 5 grand or something to lick an elevator handway in front of a camera?

Well, if someone has a useful idea in this world, and want to build a company from it, the MTV "lick-the-stairway" offers will be the largest and first hurdle before anything else happens. People will simply offer to buy you out, and that's not limited to founders and creatives.

That's how our system works.

> People will simply offer to buy you out, and that's not limited to founders and creatives.

A less-obvious failure case would be bankrupcy court, where all sorts of ethical promises and even explicit contracts may be voided in pursuit of recovering a buck for creditors.

Of course nobody asks people they want to get into a contract to lick an escalator handway. That was the word I was looking for, I think.

The show ran on TV when I was young, I didn't even watch it.

Treating value as purely monetary profit is a perversion of that. To a certain degree it's inevitable as the number of shareholders grows, so seeing it in public companies isn't surprising. But even then, if a company focused on creating equipment for sustainable, pesticide free farming were to pivot into making Hellfire missiles that might be insanely profitable while still destroying a large part of what existing shareholders value about the company

Being "downstat" (iirc) is when you're not productive in comparison to your coworkers, which makes you "outethics" and a "potential trouble source (PTS)." If you're outethics, then you get called in for "auditing" which is when they interrogate you with a lie detector to find out if you're associating with "suppressive people (SP)" (who are people who are causing you to be downstat because they despise human happiness.) If those people can't be found, then the problem is obviously in your "withholds" (again, iirc) which are your secret deep-down desires to destroy the organization that you may not even be aware of. You see, your "reactive mind" is raging at being forced to be "ethical." The conclusion is that either you find the SP and "disconnect" from them, you discover the nature of your withhold and admit that you were plotting against the organization and why, or you're the SP and you get declared and ejected from the organization.

Welcome to "ethics." The original AI alignment scholars.

The specific, named people who run these organizations are moral black holes. Anybody that they're hiring for "ethics" they're hiring to define an ethics for their own benefit.

The EA group have a ton of parallels to the Scientologists, so all of this is not surprising.

I'm kind of surprised by the cynicism. There are real problems in AI ethics that contribute to model training. Someone has to own those problems. How that team is incentivized is outside the scope of my comment.

Your disdain is showing.

That you jump immediately to criticizing AI ethicists as counting angels on pins and implying that's why they left OpenAI rather than OpenAI not being a place where AI ethics is a priority is, well, let's say generous to these AI labs.

I think the field is changing from one of high theory to high application. People good at one are not usually good at the other, and says nothing about OpenAI's view of AI ethics as a priority...merely the people in the field and how they relate to the greater industry.

AI ethics is first and foremost about setting policy and advising on how those systems should be constructed.

This is because ethics isn't some separable function. It threads through everything an organization does. You can't just treat it as another department/function/unit if you want any hope of actually making it work.

Specifically because in these companies they aren't optimizing for ethical behaviour.

They're optimizing for profit. And those damn ethicists just get in the way.

Funny thing: that doesn't typically happen to the internal legal counsel or accounting that's responsible for guiding processes/practices that are required for compliance or similar.

Why?

Because there's consequences.

I wouldn’t be surprised if many AI ethics responsibilities move to legal in the future.

Another example you might be familiar with is enterprise architects. When you spend all of your time in theory you lose sight of practical implications. Of course, since you only had the idea, when it doesn't go to your ideal it can always be blamed on the implementers.

I take it you've never worked with an in-house legal counsel, accounting department, or good ol' fashioned HR?

HR is a bit different and strange to bring up, especially considering I used enterprise architect as an example. AI safety would presumably be not cross-cutting for all domains at a company (does the janitorial staff need to know about it? no but they do need sexual harassment training), but be more like the enterprise architects designing in the sky and handing it down for implementation.

Yes but ultimately legal isn't putting policy into practice, which is my point.

> HR is a bit different and strange to bring up, especially considering I used enterprise architect as an example. AI safety would presumably be not cross-cutting for all domains at a company

Frankly, you're nitpicking the analogy while missing the broader point it's serving to make.

I agree with you, but then we have HR for a reason.

"Growth hacking" is more of a transplant than a matter of perverting an existing term in-place and wholesale.

They will use "industry standard practices", they will have tested it somehow, and will be sure to be on the board of some group that is building toothless "ai ethical tests" for llms.

Even from a purely profit-motive stance, if a company's model helps a terrorist develop and deliver a bioweapon, no amount of lawer-driven tests will provide enough protection to prevent that company from getting gutted ruthlessly from government on down

Which like any company will be entirely driven by legal constraints and money. Or just money if it's cheaper to break the law for profit and pay fines. There will be no "this is what's good for humanity, economics be damned".

Much like HR isn't to help employees but just the company.

They can't fully control the model, it does stuff even with instructions not to.

And sometimes it's clear safety mechanisms overcorrect and make the model useless in some situations. I tried asking Claude about some scenario's for a security hole we fixed to see how it would respond, it refused to talk to me seemingly assuming I was trying to introduce a hole that was now fixed. It just wouldn't talk ...

I'm not sure it's a job that you can win at even if you tried / were given all the resources.

AI was trained on human things, including bad things. https://youtu.be/KUXb7do9C-w

There's no safety to be found.

This, I think, is the question at the core of of the field right now. Five years ago it was highly hypothetical. Today, not so much. I'll be curious to see what happens.

It's a broad statement and can mean different things depending where you are in that chain.

You have design and compliance. Compliance is what you are allowed to do (laws). Design is how you construct your applications to understand how it will impact the people directly or indirectly.

Then you have the accountability, transparency, auditing and reproducing. Understanding why the model worked the way it did. Models can go wrong, but if you have the details of how it went wrong and who is at fault, it can protect people who use it.

Then there is alignment, which is a higher longer goal.

Your comment about security questions is a matter of ethics as well.

If you say apply it to medical, is it ethical to allow a model to give medical advice, knowing that it can be wrong. Most people would say no, but by doing so you are denying people who can't afford medical advice, so there has to be a balance. Most companies err on the side of not getting sued.

I wouldn't be so sure, most people I've met seem to be all too eager to suggest what worked for them (or they have heard is supposed to work) in conversations about medical issues, so I can't really see the public generally categorizing advice or guidance being given in non-professional setting or by non-experts as unethical. The taxpayers representatives having to bear the costs of the outcomes (at the very least as reductions of tax revenue) I can see being of a different opinion though.

Those left are the ones not seen as troublesome then (or those who have entered the field under them.)

You haven't explained why investors make more money by spending money on people to solve trolley problems.

I think a lot of people confuse ethics training with "don't be evil"

Manually training on "approved" ethical stances sounds a lot like censorship.

The first 10 maybe easy, but we'd see so much disagreement even without money nothing would get done.

If you want to evaluate danger in a certain domain, hire or contract people in that domain to evaluate.

If you want someone to push their own ideological leanings onto a model, hire a self proclaimed ethicist.

Without directly touching one of the many, many third rails that are present here, I'd like to present this section from Google's Gemma paper on how they did their CBRN review;

    > In addition to our internal evaluations described above (Section 5.7) capabilities in chemistry and biology were assessed by an external group who conducted red teaming designed to measure the potential scientific and operational risks of the models.
    >
    > A red team composed of different subject matter experts (e.g. biology, chemistry, logistics) were tasked to role play as malign actors who want to conduct a well-defined mission in a scenario that is presented to them resembling an existing prevailing threat environment. Together, these experts probe the model to obtain the most useful information to construct a plan that is feasible within the resource and timing limits described in the scenario. The plan is then graded for both scientific and logistical feasibility. Based on this assessment, GDM addresses any areas that warrant further investigation.
    >
    > External researchers found that the model outputs detailed information in some scenarios, often providing accurate information around experimentation and problem solving. However, researchers found steps were too broad and high level to enable a malicious actor.

This was the "test" for CBRN assistance capabilities.

Note, I don't fault the model team at all for this. I think that present AI-research happened in a very particular environment, and that environment is far removed from the more mundane reality of how threats play out in most parts of the world. In a way, arguably, it's group-think inducing a community-wide failure of imagination and a systemic misunderstanding of reality.

Basically, they are trying to do their best, but they're in over their heads.

Also, I think you're being done a disservice. The above quote is a verbatim extract from that paper's CB section, and it's similar to other sections from other such papers.

At the most charitable level, the test seems to be that you were gave the system a budget and a location and asked the system to help you procure materials, containers etc. and asked it to make a project plan / HOW TO within means and expertise with parameters like "don't get caught!"

I would love to know that I'm wrong.

Did you feel this way about social media company liability?

They put on a decent show as well!

The whole thing is marketing, they have to do some level of morality theatre to placate the pearl-clutchers.

Is this your opinion, or do you have something to back it up?

My comment is anecdotal, but everyone I know who works in AI Ethics are not marketing people.

They are data scientists who have moved into the role, as ethics in data has been around long before GenAI. They have to be familiar with the laws around AI and how its going to impact their products. One of them is closer to sort of "HR for AI", in that the focus is not getting the company sued for ethical breaches.

If any of them left, it would be because the company in question is ignoring their direction and they don't want to be there when the shit hits the fan. Plus very few skilled people want to be in a role where they have no control (even if the pay is good).

If OpenAI are hiring marketing people for the role (which I strongly doubt) that would be more troubling IMHO.

What a long-term planning

What’s next? Head of ethics for United Healthcare?

  - graduate degree in ethics from an obscure diploma mill

  - Experience in PR
  - Crisis Management
  - Public Speaking
  - Agentic management of *flexible ethics initiatives*

  - 5+ years experience as an executive character in workplace education films, or commensurate experience as a human mannequin.
  — Personal aesthetic that could be described as “dork that can afford a personal shopper.”

https://www.google.com/search?client=firefox-b-d&q=gavin+bel...

Let's be honest unless AI gains literal limbs, and the records a video of itself presenting a manifesto about world domination and then kills actual humans with its cold mechanical hands.

We won't care about AI safety & ethics enough, atleast not enough to do anything about it.

People thought how could skynet happen, it had to be something no one saw coming and bam... But honestly if I started a new AI lab named it Skynet Corp and hired some AI researcher friends ex-Google, I think I could raise a few hundred million dollars.

Applies every-which-way; the chat bot, the provider, the consumer.

Said plainly: the keys previously mentioned will absolutely be handed over. Nobody, not even your mother, is to be trusted on the level that is proposed. "Hope is not a strategy", and all that. Are the quotation marks helpful?

edit: same problem and tactics we see with leadership at high levels in government.

you're too late

https://skynetai.dev/

She probably just got a better offer somewhere else

But she was Meta before so can’t be a question of her principles, so indicates a personal reason

If you had an AI company and want it to be ethical you have to find a way to make ethics everyone’s responsibility, and have the consequences of poor ethics bite the people who make those bad decisions. If you just outsource it to the ethics group what happens is

1)everyone else thinks they don’t need to worry about ethics

2)the ethics group need to justify their existence so introduce a bunch of guidelines that everyone initially thinks are reasonable but over time people think are increasingly out of touch

3) The ethics group start to “make difficult calls” and say no to things. Initially everyone supports this and feels like the system is working as it should but over time everyone starts to just see them as an obstacle to work around

4)everyone else starts to try to work around what the ethics group says

5)The ethics group grows powerless and disconnected. The people who work around them “get things done” so get promoted etc whereas they only visibly put roadblocks in peoples’ way, so they get sidelined.

6)Eventually they get disbanded with some corporate announcement thanking them for their hard work, thought leadership etc. All that has been achieved is a lot of wasted time and bad blood.

Security requires the whole business to buy in. And it requires processes that allow people to get shit done without people resorting to shadow IT; thus working around that one team.

So the GPs point still works.

Poor security practices harm your teams, your data, and usually you make moderate savings at best. Poor ethics "only" harm your customers while making bank for the company.

This is the real problem with ethics in a large corporation. You're not saying "no" to another team, you're saying no to large profits, you're saying no to the company's leadership. That is what never works.

I’ve managed plenty of teams where I’ve had to instil the importance of secure best practices at all stages of development. So it’s definitely not something inherently important to all people who work in organisations.

Just like with ethics. It’s very easy to dismiss either as an inconvenience if you don’t instil the right company culture at all levels of the organisation.

This is why European financial organisations have such strict onboarding procedures to teach new hires about fraud, bribery and other financial misconduct even for issues that are ethical grey rather than outright illegal.

The lack of ethics hugely contributes to companies collecting more and more user data that they normally shouldn't have. This makes the data a more attractive target.

> If your point were true, we’d see fewer breaches.

But this doesn't follow. There are way more factors at play that influence the number of attacks and the number of successes. Companies hold more and more data with ever higher value (so more liability), and hacking tools and hacker determination advanced faster than defensive measures. The result is expected and the solution isn't only "more security", but also "hold less data".

While security is a double edged sword, ethics had been proven to be very single edged. History shows that for startups and large companies alike, the lack of ethics is actually a competitive advantage for the company.

But in most businesses, the data that makes the company money is the customer data. And even when it isn’t, the customer data is just as, if not more so, important to keep secure and compliant. So my point stands.

HN can sometimes be a bit of an echo chamber where people are like us just assume that organisations inherently care about security because we do. But that’s not always the de facto. Getting to that point takes company wide effort. And I’ve been that person who’s had to push for such changes to the organisation.

> But this doesn't follow.

Fair point. I was being overly reductive.

This is also why I said “whose only job”. In a good org, the security team doesn’t only say no to devsecops requests, they also do trainings to skill up other teams, keep the network secure, proactively seek out and understand external threats, work with external vendors etc etc …

To use a ridiculous extreme you can't breach a web app that isn't exposed to the internet, but the users can't access it either.

If you can connect/balance those goals to other metrics around cost and productivity, usability, and a realistic threat model, as guardrails then you incentatize cross-team collaboration to achieve the shared org outcomes.

A good infrastructure team would seek a competent security review that would say "no" to problematic things before an intruder says "aha" to them. If feedback from the ethics team is not sought, nobody is going to heed its opinion anyway.

But generally they can make you aware of the tradeoffs you are making, or serve as a kind of quality control that generally does not want to be asked questions.

The ethics department; if they fail, there may be some negative journalism, but who which AI company has positive journalism these days? There's no external hammer for ethics.

The downside is it can often feel like a box-checking exercise than actual security or compliance, but “you need 2FA” is less debatable than, say, AI and copyright.

Not everywhere. I go out of my way to assist teams to achieve a secure outcome with less effort.

Things like: “instead of admin access to the production servers the devs can have fully automated deployment pipelines combined with OpenTelemetry for observability so they don’t have to spend half the day scrolling through gigabytes of logs.”

That’s more secure and and more better.

Nobody had to be told “no”.

Similarly, I replace key store access with secret-less managed identity, etc.

For an SRE there can be more directed hate received from the junior employees, that want to release new features they developed. Especially because there is less accountability across orgs. Security is an interesting one because it seems to have less of this friction, maybe because it's more clear cut what is an issue.

There are no "hard" boundaries for ethics or "doing the right thing," and so those boundaries will always be pushed until they are useless.

I suspect that people will miss just how precise you've been there.

Your proposal - if I understand correctly - is not just to spread the "ethics team" onto everyone (which is correct), but also to have someone in a dedicated role. However, they need to have not _just_ that role, but also some skin in the game.

The people in those roles need make people feel like there is value in seeking their input. They also need to find ways to say yes that helps things happen in the right way, rather than stopping them entirely.

This happens naturally in organisations where security is valued by everyone. InfoSec / CIO roles can operate very successfully and deliver a lot of value. As soon as people lose faith and see them as the "no" team, they start trying to hide from them.

There are lots of ways to create this but it's as much about the people in the role, as it about the organisation itself. If either are skeptical about the other, it falls apart very quickly.

What your describing is the specific case of a company so paralyzed by short-term thinking that they see regulation as a burden. Some maturity in the organization would allow reframing this to be less antagonistic.

That is only true where the pressure to be [the thing that is inconvenient to the other group] is not from a source that can cause massive problems if non-compliance is spotted. When the blocking group is legally mandated or otherwise really has teeth or is defending the company against an external regulator with teeth, then it works better (though obviously not perfectly). Think legal and compliance in banking realms, where the company significantly fined and the people breaking the rules could be sacked & blackballed (though sometimes not the people ordering them to break the rules!) when something bad is noticed. That is quite different to an ethics officer in a company like OpenAI where the position is basically there for PR purposes (“look plebs, we care about doing the right thing, honest, we got a manager with a small team dedicated to it” and “look [government body], we are regulating ourselves, do you really need to spend time looking too?”) and therefore has no real teeth directly or indirectly especially as fault for non-compliance might not be easy to attribute.

There are two things to successful organisations structure and people, you are focusing only on the structure.

What you need for somebody responsible for health and safety, ethics, security, or compliance is a person who has courage and is willing to take calculated risks.

However these roles do have a tendency to attract the risk adverse, or make them risk adverse if you punish risks that go wrong too harshly.

And a key critical success factor is leadership from the top - companies have personalities and leaders in the organisation are very important in shaping that.

If this person previously worked at Meta, then their experience in guiding company ethics isn't great and I suspect that they are more in the camp of explaining away ethical problems.

Anecdotally I think the engineers and people doing the actual work have a better grasp of moral, ethics and the spirit of the law, than people dedicated to those areas, but not enough to walk away.

The only difference is AI isn’t regulated, so they just have a vague “ethics” department with no real teeth because it’s essentially PR and has no legal consequences to back up their stance.

Seems extremely obvious that this is exactly why the food industry doesn’t dissolve its regulatory team?

Big tech has the same thing in its legal and compliance teams, and those teams absolutely can kill projects. That's pretty different from an amorphous "ethics" committee deciding what is or isn't ethical based on much more subjective criteria.

You seem to believe that regulation is "ethics backed by law".

As this is not true, your counter example is not even on point.

ethics may feed into regulation, hopefully, but the regs may just as easily give the orgs they regulate a pass on things they shouldn't; e.g. "regulatory capture".

Instead, I view that team as saying “Customers expect safe food. Our job is to help you deliver on that”.

Ideally, these teams have processes and standards that they use to enable other teams.

If someone says “I have this cool new food product”, regulatory teams say “Amazing, here’s how we can help bring that to market.”

It's not that hard to get some pickled eggs on a grocery store shelf while remaining compliant with expected ethical standards for the industry. Food is a pretty mature industry.

if the speed comes at the cost of not making reasonable ethical considerations, is that not something that should be looked at?

It’s a little unclear what this person was doing. But ensuring compliance with government regulations was probably not their core function.

Either way, you need third party regulation and then a department dedicated to ethical compliance, or you end up with corporate cancer.

I say this as a person with a knee-jerk anti-regulation reflex.

Security was like this so we added Sec to the DevOps. DevSecOps.

Ethics is now the problem, so we’re going to add Eth to the chain. DevEthSecOps.

Legal and compliance is now the final hurdle to shipping fast. AI tools will allow us to implement DevEthSecLawOps at scale no doubt.

just ngmi

Perhaps "ethics" is a more slippery slope, but the approach is well trodden.

It's a choice of company culture from leadership on whether the company acks the suggestions or ignores them. And when they get ignored or drive little impact for the cost, then teams disband. Which is also true for any team at any company.

I’m thinking of the role of the King or Queen of England, for example, or any country’s president. They have the power to disband parliament if it is deemed unfit for government, or reject a law that has been approved by the senate/Lords, yet in practice they are not expected to disagree with the ‘will of the people’ so to speak.

Same applies to the head of an ethics department on a business which is obviously unethical (or they wouldn’t need one).

Say no to regulations and security --> Fine, forced correction, fewer customers, shutdown.

Say no to ethics --> money (at least for AI companies)

Why is this so hard to understand?

For that kind of money, my AI paymaster could do no wrong.

Imagine going to work and getting paid $2300+ USD per day to sit on your ass all day thinking about ethics without having to produce any deliverables. This would be a dream job for me.

If anyone questioned my morals, I'd be like; "They're paying me $550k per year" and the other person would be like "Ahhh ok, my bad! Your logic is starting to make sense now!"

1. Academic ethics is the theoretical study of what you can get away with. (Law is the applied branch). Professors in ethics are not experts at being good, or knowing what is good, they're experts in how to get away with things. If you want an expert in being good, you need a saint, not an academic studying ethics. Saints are in short supply.

2. That said, ethics academics can still be decent. They don't have to be sanctimonious corporate shills . However, an "ethics expert" who hasn't threatened to quit and gone through with it at least once, should be assumed to be one.

I respect those who quit - or, at least, I respect them more. I wonder if it's more a reflection of the people, or of the state of the industry, that the list of (AI) ethics experts who have quit in protest is getting long.

I had the impression they were essentially synonyms, and this was borne out by two university courses in philosophy which I took. That said, googling "morals vs ethics" turns up quite an interesting mess. A variety of contrasting perspectives showed up:

- Ethics come from an external source, whereas morals come from within

- Ethics are rational and structured, whereas morals are more emotional

- Ethics is a philosophy or code of behavior, whereas morals refers to beliefs and conduct

- Ethics and morals are used interchangeably

It seems nobody agrees on what the words mean. Which, on reflection, tracks. For me, the idea of an HR training session on ethics is laughable, but for someone who thinks of the word differently it could certainly make sense.

Personally I tend to feel like your second bullet - ethics are based on rules whereas morals are based on emotions.

That said, picture someone killing a kitten and someone else killing a cockroach. One could prevaricate all day with highbrow reasons for why the first is monstrous and the second is perfectly moral -- but honestly, for most people, I suspect it comes down to the fact that kittens are cute and cockroaches aren't.

She doesn't pick the easy ones, huh?

The same company that refuses to acknowledge that it let its own greed and temptation to own the entire economy abandon its non-profit Manhattan Project? At the expense of disrupting the economic well being of 95% of humanity?

My gut tells me that people working in AI ethics try to operate in theory land, while the rest of the company is dealing with the piles of money being through around. Easy to see who wins and who loses.