If you were being paid $300K/year, it costs your employer (as a rule of thumb) 1.5x, or about $450K/year. The only reason they could spend $450K/year on you is if they expected to make at least that much money with you, compared to what they would make without you. Even if you aren't directly working on this stuff, you are providing them with capital that is being spent on it.

Arguably they don't have to make that money off you directly.

They just need you to not be elsewhere making that money for someone else, or disrupting a market that they are entrenched in. IMHO much of the FAANG hiring / head count / acquisition process could be analyzed in this light.

It’s good that you would refuse to do the work. Sadly they can easily find someone who badly needs the money, doesn’t understand the problems the feature creates, or doesn’t care about doing the right thing. If all the workers organized it would make a bigger difference, but the company actively discourages worker organization.

by "like that" do you mean "a community moderation system" or "a system for the CCP to backdoor explicit takedown rules into the system?"

Based on what we're seeing right now, this is likely caused not by the latter, but by the former. Consider: the ML-assisted thread moderation logic can be vulnerable to brigading. If several tens of thousands of Chinese people decided to start flagging comments with that phrase, YT would also start killing the phrase (because its sample is biased towards seeing "That phrase usually results in a flag, so the community clearly doesn't want it").

I like to think I’d take the high road but the Milgram shock experiment[1] shows that most people, including me, probably wouldn’t.

[1] https://en.m.wikipedia.org/wiki/Milgram_experiment

Then we need to come up with ways to counteract the Milgram effect, if that's even possible.

Yes indeed. It’s one of those spiky problems with human nature.

It ranks up there with the problem of dealing with seductive disinformation.

For everyone pointing out that nothing happened in Moab there’s someone else with a Remember Moab bumpersticker[1].

[1] Neal Stephenson reference.

No one will ever ask you to implement something "like that". It probably was an ordinary blacklist that probably started to "protect children" form porn and such, then it went on to "protect teenagers" from radicalization, and now it will protect adults from anti-communist/chinese comments. We've made the bed.