> these are earnest people.

I agree.

I'll also add that when my startup got acquired into a very large, well-known valley giant with a sterling rep for integrity and I ended up as a senior executive - over time I got a first-hand education on the myriad ways genuinely well-intentioned people can still end up being the responsible party(s) presiding over a system doing net wrong things. All with no individual ever meaning to or even consciously knowing.

It's hard to explain and I probably wouldn't have believed myself before I saw and experienced it. I'm pretty sure I ended up moving on before I compromised myself but preventing that required constant vigilance and led to some inter-personal and 'official' friction. Standing against an overwhelming organizational tide is stressful and never leads to popularity or promotion.

Someone posted this here the other day and it uses _Demons_ to discuss exactly your point.

https://possessedmachines.com/

Wow. Only one page in and already bookmarked to absorb later. Thanks for the link.

That's why we're setting up adversarial benchmarks to test if they are doing the thing they promised not to do, because we totally trust them.