> Jeff was micromanaging her citation list, which is almost unheard of

This is quite normal, even for purely technical papers [^1], and absolutely important for an expository paper like this one. Narratives are always formed by selectively including references.

> the entire AI community is now on her side

On the MachineLearning reddit, which is popular among a proportion of ML grad students at least, it's completely different. Indeed the top comment in the discussion thread there [^2] discusses this discrepancy. And the fact that I'm creating a throwaway account for this reply is telling.

[1]: The review process of ICLR 2021, a premier ML conference, is taking place on public at https://openreview.net/group?id=ICLR.cc/2021/Conference. You can navigate through the reviews and see how many requests are wrt citations.

[2]: https://www.reddit.com/r/MachineLearning/comments/k6467v/n_t...

>Timnit responded with an email requiring that a number of conditions be met in order for her to continue working at Google, including revealing the identities of every person who Megan and I had spoken to and consulted as part of the review of the paper and the exact feedback. Timnit wrote that if we didn’t meet these demands, she would leave Google and work on an end date. We accept and respect her decision to resign from Google.

https://twitter.com/JeffDean/status/1334953632719011840

With regard to the internal review thing, I thought this comment made a good point: https://www.reddit.com/r/MachineLearning/comments/k6467v/n_t... Sure, maybe Google is usually lax with internal reviews. But if you're going to write a paper which says "this area of research Google is engaging in is harmful", and neglect to mention the fact that Google is also doing research trying to address the harms, then from Google's perspective, you are just smearing its corporate brand without making any real progress.

The weird part is, the paper didn't seem to be saying that. It was mostly about energy usage, and bias in language models. The reviewer described it as "anodyne," which seems apt: https://old.reddit.com/r/MachineLearning/comments/k69eq0/n_t...

I'd totally agree with you. That would indeed be ridiculous. But... It's strange... each time a new argument pops up, I dig into the new detail, and surprise: it seems to have a straightforward, boring answer. "This is a pretty standard paper. Google wouldn't have been hurt reputation-wise by letting it through. And we should probably be thinking more about energy usage and bias. She didn't namedrop all relevant research, but there doesn't seem to be anything here to demand a retraction over."

It only gets stranger when you take this into account, too. From the journal reviewer:

However, the authors had (and still have) many weeks to update the paper before publication. The email from Google implies (but carefully does not state) that the only solution was the paper's retraction. That was not the case.

In some parallel universe, Google could re-hire her, Jeff and her could sit down and hammer out the paper, send the updated version, and there would still be two weeks to make even more edits. Isn't the point of the edit window to address these problems?

What really got my attention though, was that she informed everyone months ago that her and her coauthors were writing this paper. She wasn't working on some hit piece of a research paper. It's just ... a standard survey of the current ML scene circa 2021. I read the abstract and go "Yup, we use a shitload of energy. Yup, we should have better tools for filtering training data -- I've wanted this for myself. Where's the bombshell?"

For all the fuss people are making, you'd expect the paper to be arguing that we should stop doing AI for the betterment of humanity, or something weird. But it's nothing like that.

The entire course of action you suggest could have happened were not for Timnit going public by her own volition, accusing her employer and coworkers of unethical behavior in the process, and encouraging sympathetic colleagues to apply external political and judicial pressure on Google. What for, so she can publish a review paper disregarding relevant internal feedback?

She made a lot of fuss and burnt a lot of bridges. Nobody forced her to do so.

> Timnit wrote that if we didn’t meet these demands, she would leave Google and work on an end date. We accept and respect her decision to resign from Google.

This means the date of her leaving was not set in stone, ie both parties would agree to the date or a date when she would leave. She had a vacation coming up and they would discuss it when she returned from vacation.

>According to Ms Gebru, Google replied: "We respect your decision to leave Google... and we are accepting your resignation.

>"However, we believe the end of your employment should happen faster than your email reflects because certain aspects of the email you sent last night to non-management employees in the brain group reflect behaviour that is inconsistent with the expectations of a Google manager."

She was fired, not on account of the content of her paper, but the way she expressed her feelings to her colleagues, which wasn't an exhortation for them lay down their tools, but because laying down their tools made no different as their work was ignored.

She was not fired. She offered to resign if her demands were not met.

The only difference is that the timeline was not of her choosing. She knew exactly what she was gambling for when she wrote those words. Why would Google want someone around who doesn’t want to work at Google? Better to accept their resignation and let them go immediately. It happens all the time. I once gave 2 weeks notice and my boss said I might as well leave immediately since there was no point in hanging around (I was consulting and on the bench). It happens.

If they decided the date of the "resignation" without informing her then they fired her. That is pretty obvious.

And it wasn't because of her demands concerning the paper, it was on account of the way they interpreted of an email she sent to her colleagues.

> Because, as far as I can tell, the entire AI community is now on her side

Oh because how dare people judge. You would be instantly taken to court of public opinion for talking down a black female, AI, ethics, researcher.

If people have opposite opinion, they would keep it to themselves. But saying the entire AI community had rallied behind her is wrong.

I've been hoping to address that point, actually. Because, I get where you're coming from, and I've felt those frustrations myself. Let's just say I'm probably the last person to hop on the "do what I say or else" bandwagon.

... but like everything else, that turns out not to be an issue here. I would feel fine saying what you said. It's nothing to do with the color of her skin, or the fact that she was working on ethics rather than optimizers. Horrible employees who cause drama wherever they go are a liability and a downer, and I'll happily say that to whoever's listening. Black or white or purple, the goal is to serve the company's business interests.

But, much to my own surprise, as someone who grew up memeing on 4chan and poking fun at leftist dogma, here I am after about a year and a half of ML, wondering "Where's my harassment? I was promised harassment."

Because I feel exactly the opposite. Not only is everyone in the ML twitter scene cool, but they're some of the most open minded people I've ever met. Sure, you get some people showing up sometimes to give you a hard time, e.g. when we train danbooru: https://i.imgur.com/RMZd6mu.png and then say that the dataset is very objectifying, and so on.

But the antidote is to simply be straightforward. Make it clear you actually hear them. "Yes, I am concerned about that, but it's simply the problem domain; people use danbooru as a repository of this content. You're right that we could e.g. make a classifier to pick out the cooler looking gals and focus on those, but the challenge is simply to solve the problem at all. Once we have lovely auto-generated anime, it'll be straightforward to filter it. Come help us get there! It's fun!"

With all the horror stories I've heard, and how afraid everyone is of "the mob," I went into it expecting to "smile, and don't tell them what you're thinking." And I was really surprised to find exactly the opposite atmosphere. Everyone pretty much agrees that yeah, we have a bias problem, and that it's probably good to address that. People also seem to agree that it's ridiculous to take it too far, e.g. when OpenAI enforces a mandatory content filter that you can't turn off and isn't too choosy about what it deems harmful, and prevents you from shipping to production unless you enable it.

So I'm just scratching my head like ... "am I the baddie?" https://www.youtube.com/watch?v=hn1VxaMEjRU&ab_channel=roots...

Because I'd be a part of the problem, if you felt like you can't talk to me freely (at least in private). I don't want to foster that kind of environment. So my reaction of the idea of you "talking down a black female, AI, ethics researcher" is "Y'know, I see where you're coming from, and I was worried about the same thing, but I'm just not getting that sort of vibe at all. I was surprised too; I expected the opposite."

And once you relax about it and look around, the most interesting part to me is that you start to feel like "well... why look at whether she's a black, female, AI, ethics researcher? I should probably read her work and listen to what she's saying, and judge for myself whether it seems crazy." And when you sit down and really listen to people, and put your full mental focus on what they are saying, I find it hard to disagree with a lot of their points, simply from a logical perspective.

So you might argue "Well, you're just part of that culture then. You'd be appalled how I feel, but I'll keep that to myself." Fair. But it's so weird being in a situation where everyone is like "watch out for that mob!" and meanwhile all of the people who actually work in the field seem pretty cool to me.

Everyone feels equally lost, i.e. that we have this magical new power (ML) that we don't really know what to do with. We know it will affect society, but we don't know how it will affect society. We also don't know the best ways to guard against some of the obvious problems on the horizon.

I ran into that problem myself. Since I'm already on a ramble, I may as well lay it all out, because it really is interesting: I was training some FFHQ latent directions, trying to get a skin color working. And I ended up discovering, quite by accident, that my latent vectors were "racist." My model was generating caricatures of black people that I would not be comfortable showing. And I couldn't figure out why. Why black people? I flip the skin color to white, no problem. Flip it to asian, no problem: https://twitter.com/theshawwn/status/1184074334186414080

Flip it to black, and horrible results. (I mean "horrible" as in "this would be an especially bad idea to show anyone," rather than merely "it has some visual defects.")

The answer to this mystery was obvious, but only in hindsight. There are far fewer black people in FFHQ than whites or asians. I found a classifier, got it to identify an order of magnitude more black faces than I had before, retrained my model, and the result was instantly so much better: https://twitter.com/theshawwn/status/1209749009092493312

That experience stayed with me to this day. I think about it a lot, because it would be so easy to overlook that kind of bias when it's numerical data rather than facial data. And as far as I can tell, that's exactly the sort of ethics that Timnit has been arguing for: we need to pay more attention to bias, and unexpected ways that bias can creep in. Which seems reasonable to me.

I don't really know why I'm posting this to you, but, just in case it changes your mind, I leave it to you. I really thought I'd end up feeling every bit as pinched as you expressed, yet it's nothing like that. Felt quite the opposite. I keep re-reading http://paulgraham.com/orth.html wondering if I have orthodox privilege, or if my social group simply doesn't include people who are comfortable enough to express themselves around me, or what. Because you say that there is a massive number of people who feel exactly the opposite, and I can't help feeling curious where they are. Our discord's up to 1200 people, and they don't seem to be there. I talk to dozens of researchers, sometimes on a weekly basis, just to poke my head in and see what they're up to. They don't seem to be there either, even in private. And, picture someone who is the opposite of "leftist" in basically every way. I know a few people like that in the ML scene, and even they don't seem to be saying "whoa, this bias stuff has gone too far, and these ethics concerns are nonsense." It's the opposite. Eleuther has a dedicated ethics channel, people spend a lot of time freely debating about what the "right" ideas might look like, and so on. We also maintain internal research channels with the idea that, if people have concerns of the type you mention, they can freely express themselves there with no fear of any kind of retribution -- that's the whole point of having secret research channels. It's up to ~30 or so active researchers, and no one has brought up concerns like that.

So you see, I end up being dragged to the conclusion that, yes, the politics / activism stuff is a concern, but no, it doesn't seem to be affecting anything. We're not hearing "do this or I'll bite your face off," or something nuts like that. It's more like "Could you please listen to me for a bit? I have this experience I'd like to share." And the experiences tend to be interesting, at least to me.

So that's why I urge you to be super skeptical about the angle you mention. ("You'd be instantly taken to court of public opinion...") Dig into the situation and look for evidence of that yourself. Don't pay attention to newspapers; talk to researchers, and ask them how they feel about it. I just can't find any trace, no matter how hard I look. I feel if you also look, in a scientific way, for evidence to support your concerns, that you might not find it either.

Anyway. I related to what you were saying and just wanted to give perhaps a new way of looking at it, since it's what changed my mind. Feel free to hit me up in twitter DMs if you're looking for someone to chat with about some hard topics, since those are often the interesting ones.

> The answer to this mystery was obvious, but only in hindsight. There are far fewer black people in FFHQ than whites or asians. I found a classifier, got it to identify an order of magnitude more black faces than I had before, retrained my model, and the result was instantly so much better: https://twitter.com/theshawwn/status/1209749009092493312

Ironically, that exact observation about that exact dataset was what got Dr. Gebru so mad at Yann Le Cun in the twitter thread that people brought up. So maybe you're just lucky with who saw your tweets.

You've worked with a visual appearances dataset, lacking sufficient examples from one class of entities, and it failed to perform well for that class. You solved the problem by adding more example of that class. While the malfunction might have hd some, as of yet unquantified, real world impact in some hypothetical police face recognition system, it doesn't follow that:

a. Datasets that are not about visual appearances, are prone to the same problem and to the same degree. Perhaps the house lending datasets / systems have small race (visual appearances) issues, but large class issues. The political debate of how to handle class issues is as old as politics have been around.

b. The extent of real world impact, which depends on the actual system deployed. Perhaps a hypothetical real world system has a 1% failure rate vs .1% failure rate. Should we stop developing useful system just because they are not produce exactly the same results across all visible demographics we can carve ourselves into?

c. Can the impact be mitigated by human post processing. The hypothetical face recognition system is part of the judicial process, there are many checks and balances before one gets to suffer drastic consequences. For example a human actually looking at the picture, or a solid alibi. "Your honor, I was skiing in Canada at the time of the alleged Florida murder".

As others have expressed in this thread, dealing with first order visual issues is easy. everyone can agree at a glance what a correct solution to visual questions is, and bugs are usually straightforward to fix. Language issues on the other hand, are second order, everything is subject to interpretation. Once we open the can of worms of talking 'critically' about language and AI, we are getting uncomfortably close to language police, and via Shapir Whorf, to thought police. The BIG underlying stake of 'AI Ethics', one that possibly neither side has completely articulated just yet:

Should a small group (in the thousands) of hyperachieving hyperpriviledged individuals working in the AI labs at the handful of megacorporations controlling the online flow of human language, get to decide what we can speak, and by extension what we can think?

Re the existence of an internal review (tl;dr: it definitely does exist!): A Google employee here who's actually published research there. There definitely IS an internal review and an approval process and every single paper I have published while there had to go through it. Everyone I knew had to submit for an internal review well in advance and I personally couldn't even submit to a conference review once because we only asked for the internal review 3 days in advance, which wasn't enough time. My understanding was that everyone around me was adhering to this.

Thank you for speaking up! To be clear, there’s no question it exists. But multiple (admittedly former) google researchers have gone on record saying that this is meant to be a quick scan for trade secrets leakage, and nothing more than that. Certainly not some kind of academic integrity review. That’s what the journal reviewers are for.

The point of the scientific process is to be allowed to fail and to be mistaken. And the paper was pretty bland. Sure, she didn’t namedrop all relevant research over the last decade, but why demand a retraction? Especially when the reviewers posted on Reddit that there was a big edit window to make revisions. Adding a few more cites seems like “oh, why not throw in X?” then you both go out to lunch / email an updated version later that day. A retraction is basically “your last few months have been wasted,” right? I’d probably be upset too.

Thanks again for the datapoint about the 3 day window. It seems rather selective, but that’s standard for any bigco. I’m just having a hard time stomaching the idea that your ideas need to pass through some anonymous internal review panel where you don’t even know who’s judging you. Is that really what it’s like to be there? Seems strange.

Typically this is not something you worry about too much. You work on a research project, you talk to people around you, your research manager and everyone around you pretty openly, and if something was going very wrong, you'd likely know because someone would have told you along the way. So by the time you are submitting for internal review, the expectation is that you will almost certainly pass, but typically the reviewers tale their job pretty seriously (and so did I when I served as one) and spent a week or two reading through your paper in their "spare" time and writing a comment on the science, the method, the conclusions etc. The not-leaking-secrets part is certainly there, but that's the lowest bar to pass, so one rarely worries about it practically if one works on public datasets and doing fundamental science.

My experience might be different from what Dr Gebru was going through since I never rubbed against anything that could have been considered company secret. My work was entirely academic and I never felt that I was restricted in any way in the questions that I could ask or the papers I could write. That is likely very different when you are criticizing a product, using internal data etc, which might have been the case with her. It also seems that she was in no way diplomatic about her actions.

When you do fundamental research there, it's as free or possibly even freer than standard academic institutions. As I said, I personally never felt any implicit let alone explicit forces telling what to work on / what to avoid.

The fact is that you’re wrong. There is an thorough internal review process. It’s not just a rubber stamp. So please stop spreading misinformation. What Dr. Gebru went through was standard.

What organization are you in? Several people have said AI and Brain don't work that way. Someone from Tech Infra said their papers could get spiked just for not being interesting enough.

Brain

The question is whether the review process vets the topic of the paper, the writing, the citations etc., or it's only a screening that avoids revealing company secrets.

The paper sounds boring, though.

She hits some boring bureaucratic resistance, lost her shit, called everyone racist and gave ultimatums. She chose this hill to die on.

If she were uncovering some major ethical scandal, I'd be on her side, but this?