Microsoft's old chatbot, Tay, would learn by talking on Twitter. She met some people with...some things to say. And learned from them. In 16 hours she had learned so well they had to shut her down.

That failure mode was obvious to every single human person in the universe not working on Tay. (And probably a few who were.)

I think we need to be realistic and accept that people are going to pick the expert that agrees with them, even if on paper they are far less qualified.

She's a professor of computer linguistics; it's literally her field that's being discussed.

The list of signatories includes people with far less relevant qualifications, and significantly greater profit motive.

She's an informed party who doesn't stand to profit; we should listen to her a lot more readily than others.

How are fame, speaking engagements, and book deals not a form of profit?

She's intelligent and worth listening to, but she has just as much personal bias and motivation as anyone else.

The (very small) amount of fame she's collected has come through her work in the field, and it's a field she's been in for a while; she's hardly chasing glory.

People don’t have to be chasing fame to be warped by it. She has cultivated a following of like-minded people who provide ever more positive feedback for her ever more ideological writing.

I mean she is literally dismissing people who disagree with her based on their skin color. Can we stop for a minute to wonder about the incentives that encourage that?

(and I generally like her writing and think she has interesting things to say… but I do see a reward cycle going on)

Her field has also taken the largest hit from the success of LLMs and her research topics and her department are probably no longer prioritized by research grants. Given how many articles she's written that have criticized LLMs it's not surprising she has incentives.

LLMs are in her field; they are one of her research topics and they're definitely getting funding.

We absolutely should not be ignoring research that doesn't support popular narratives; dismissing her work because it is critical of LLMs is not reasonable.

In her field doesn't mean that's what she researches, LLMs are loosely in her field but the methods are completely different. Computational linguistics != deep learning. Deep learning does not directly use concepts from linguistics, semantics, grammars or grammar engineering, which is what Emily was researcing for the past decades.

It's the same thing as saying a number theorist and a set theorist are in the same field cause they both work in the Math field.

They are what she researches though. She has published research on them.

LLMs don't directly use concepts from linguistics but they do produce and model language/grammar; it's entirely valid to use techniques from those fields to evaluate them, which is what she does. In the same vein, though a self-driving car doesn't work the same way as a human driver does, we can measure their performance on similar tasks.

Hmm I looked into it, and looked at papers/pdfs in google scholar's advanced search with her as an author that mentioned LLMs or GPT in the past 3 years. Every single one was a criticism about how they couldn't actually understand anything (e.g. "they're only trained on form" and "at best they can only understand things in a limited well scoped fashion") and that linguistic fundamentals for NLP was more important.

Good to know my hunch was correct

It is not that she is critical of LLMs that is the issue.

Instead, it is that she has strong ideological motivations to make certain arguments.

Those motivations being that her research is now worthless, because of LLMs.

I don't believe the alignment doomsayers either, but that is for different reasons than listening to her.

She's the first author of the stochastic parrots paper, and she's fairly representative of the group of "AI safety" researchers who view the field from a statistical perspective linked to social justice issues. That's distinct from the group of "AI safety" researchers who focus on the "might destroy humanity" perspective. There are other groups too obviously -- the field seems to cluster into ideological perspectives.

Current topic aside, I feel like that stochastic parrots paper aged really poorly in its criticisms of LLMs, and reading it felt like political propaganda with its exaggerated rhetoric and its anemic amount of scientific substance e.g.

> Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.

I'm surprised its cited so much given how many of its claims fell flat 1.5 years later

It's extremely easy to publish in NLP right now. 20-30% acceptance rates at even the top end conferences and plenty of tricks to increase your chances. Just because someone is first author on a highly cited paper doesn't imply that they're "right"

>She's the first author of the stochastic parrots paper,

That alone is enough to disqualify any of her opinions on AI.

She’s contributed to many academic papers on large language models and has a better technical understanding of how they work and their limitations than most signatories of this statement, or the previous widely hyped “AI pause” letter, which referenced one of her own papers. Read her statement about that letter (https://www.dair-institute.org/blog/letter-statement-March20...) or listen to some of the many podcasts she’s appeared on talking about this before making presumptions uneducated statements.

Thanks for the link, I read it with interest but philosophically it is a flawed argument. It's a nonsequitur or something, for the following two reasons.

First, I'm inclined to think that longtermism is an invalid and harmful ideology, but also acknowledge that AGI / existential risk is something that needs looking at seriously. The external factors, such as corporate interests and 1% wealthy interests/prejudices, are not a good reason to dismiss AGI concerns. I'd like to imagine there's a reasonable way to address short-term issues as well as long-term issues. It's not an either-or debate.

Second, even from just a reading comprehension level: one side says AGI is a problem, then the other side cannot just say, "No, AGI is a false problem and here a the real problems". The reasonable argument is to say, AGI is a false problem because <of such and such reasons>. Bender et. al are just sidestepping the moot point, and rhetorically this is not an okay move. I think honest experts could simply say, ultimately we don't really know what will happen. But that would be boring to say because it would require acknowledging multiple issues being valid.

So I think this is actually an example of different factional experts talking past each other. It's funny that famous intellectuals/experts constantly do this, let their egos get the better of themselves and having a real intellectual conversation rather than make basic debate mistakes like nonsequiturs that any college student should be able to point at.

> First, I’m inclined to think that longtermism is an invalid and harmful ideology

It is, but (aside from as a sort of sociological explanation of the belief in AGI risk) that’s mostly beside the point when discussing the problems AGI x-risk. The problem of AGI x-risk is that it is an entirely abstract concern which does not concrete flow from any basis in material reality, cannot be assessed with the tools of assessing material reality, and exists as a kind of religious doctrine surrounded by rhetorical flourishes.

> The external factors, such as corporate interests and 1% wealthy interests/prejudices, are not a good reason to dismiss AGI concerns.

They are way of understanding why people who seem (largely because they are) intelligent and competent are trying to sell such hollow arguments as AGI x-risk. They aren’t, you are correct, a logical rebuttal to AGI risk, nor are they intended as that; the only rebuttal is the complete absence of support for the proposition. They are, however, a tool that operates outside the realm of formalized debate that addresses the natural and useful cognitive bias that itself is outside of the realm of formalized debate that says “smart, competent people don’t tend to embrace hollow positions”.

> Second, even from just a reading comprehension level: one side says AGI is a problem, then the other side cannot just say, “No, AGI is a false problem and here a the real problems”.

1. If they couldn’t, it wouldn’t be a “reading comprehension issue”, and

2. They can, for the simple reason that there is no material support for the “AGI is a real problem” argument.

> Bender et. al are just sidestepping the moot point,

A point being moot in the sense that AGI x-risk is is a reason to sidestep it. (The danger of using auto-antonyms.)

> I think honest experts could simply say, ultimately we don’t really know what will happen.

To the extent that is accurate, that is exactly what the Bender/Gebru/Mitchell group does. The problem is thinking “we don’t have any information to justify any belief on that” is one sided and means that utility of AGI is somewhere between 0 and the negative infinity that the x-risk crowd calculates from (some unspecified non-zero finite probability) times (infinite cost), whereas the reality is that we have as much reason to believe that AGI is the only solution to an otherwise certain existential calamity as to suppose it will lead to one. The utility is somewhere between positive infinity and negative infinity.

A point being moot is a reason to agree to disagree, if they can't agree on the premise. But they need to say that. If I were writing a letter, I would say it because that's just being sensible.

This isn't about logical debate. This is about reasonable, non-sophistic writing at the college level or higher. And there are basic standards, like if they don't know the future then they must explicitly acknowledge that. Not rhetorically "do that" in the essay. Literally write it out in sentences. They didn't.

I can think of 3 examples where such explicitness was done. Chomsky's letter gave explicit reasons why AGI is a false issue (and he was pilloried for it). My computer science professors literally, in their deep learning class and in their theoretical machine learning research seminars, have literally acknowledge that we don't know almost anything about the fundamentals nor the future. That scientific humility and level of intellectual conscientiousness is needed. That is absent in this discourse between the experts. And note, by that, I also include the 22-word "letter" which doesn't actually explain why Hinton and the rest of the signatories think AGI is an existential risk, what their specific reasons (your "material evidence") for that are.