From my experience with a very large bureaucracy, I believe the answer to this is "the worker bees don't believe this; the requirement to implement automated moderation [in this way] is imposed by managers and/or executives" - and speaking up can get you retaliated against.
- CEO has a goal of getting support ticket resolution times to X
- Some exec is tasked with flagging content, and allowing very little unapproved content through (creates lawsuits)
- Some exec has a budget of Y for resolving support issues
- Some other execc else wants to get a higher level technical win on their resume
Then in total you get someone who builds an automatic flagging system. This creates support tickets. You need to drive them down, but you only have Y budget to do so (so you can't hire an army of high-quality reps). You hire a smaller amount of reps on a budget. They do a bad job from being undergunned; someone builds an internal product that will magically help them resolve issues faster but it basically just helps them resolve issues really fast by identifying the obviously innocent and denying appeal on most cases. At the end of the day an understaffed team closes a huge # of tickets without prejudice. But at no point did anyone say "F those users."
This is actually worse than some tone deaf exec who hates users, because you can't solve it by replacing someone bad.
I can verify this is true since I used to work there.
https://www.bloomberg.com/news/features/2021-01-29/amazon-ga...
They are not broken approaches. They are optimizing for net income. Higher net income for the company means higher compensation for individuals at the company, since they are being paid with equity. Spending more money on labor means less net income for the company means lower compensation for individuals.
- They are extremely self-assured of their competence and the effectiveness/completeness of anything they design.
- Google has nearly two-decade-culture of drive-by product development. People work on something, it becomes a product, the laser pointer of Shiny dances nearby, and they're off to work on something else and the product matures at a glacial rate, if at all.
They also don't care. Money printing machine go brrrrrrrrrr to the tune of $10BN/year.
Can be awful for users, but hey, the company lives on.
My point here is that the problem is not entirely technical. You can't make ideal model without false-positive. You just try to optimise it by using discovered cases like this as input to bake better one. So the only thing to blame here - not enough human operators. Thats why you probably don't get any answers here from google engineers. The root cause either not technical, or too technical to comment it outside.
Hiring individual people to address your issues is going to be hard to scale in a way that allows a free service to be rolled out to billions.
I don't work at Google, do work at another company that I have no doubt has the same issue, but not discussing in the context of that company either.
But what I never see is any of those Googlers taking the time to explain why they feel these seemingly-broken approaches are the only ways to solve the problem of moderation at scale. To me, it seems insulting to one's intelligence to receive an automated message that your channel's been banned, be given the option to appeal, and to receive an equally automated message rnd() minutes later that your appeal has been denied. But what do I know? I've never been tasked with solving such an engineering problem.
I know FAANGers here may feel put upon by what appears to be a hostile environment against their companies, but that's at least in part because you've never said exactly which technical and practical challenges lead to this state of affairs. If it's not against your company's NDAs, I challenge you all to try; whether the community agrees with you or not, I think we would all learn something useful from a productive discussion about the problem.