I think what the conclusion, though, is that the loss of privacy to language models pales in comparison to the loss of privacy to the rest of the internet.

The language model is just a different way of organizing the information that is already out there. It doesn’t create or expose any new information, and it’s ability to correlate information with other information isn’t superior to the other, existing, ways. If someone is attempting to find out the real person behind your username, a google search still works better than typing your username into the language model.

The real lesson (and one that predates language models like GPT), is that we can no longer rely on the strategy of being ‘lost in the crowd’ to preserve our privacy. When you took all of those actions that connected your username to your real name, you probably thought, “this is just one little obscure thing, no one will ever find it.” Of course, Google finds everything.

This is completely unrelated to things like GPT.

> The language model is just a different way of organizing the information that is already out there. It doesn’t create or expose any new information, and it’s ability to correlate information with other information isn’t superior to the other, existing, ways.

At some previous logistics company this was something I questioned regarding a new tool for managers that could more easily track driver performance as opposed to digging in many old systems, taking notes and making inferences.

Because the new tool made it easier, even though the data is the same, the company's lawyers were very skeptical of this in relation to the driver's privacy.

There is something grey about this I think. Ease of access do matter to some extent, even though my engineering brain says the data is the same so it doesn't matter.

I actually completely agree with this.

A lot of our historical privacy has been based on practicality. We don't worry about someone monitoring us all the time in public because it would be expensive and exhausting to follow us around all the time, and there is no way to follow everyone.

Computers change this. Computers are great at doing repetitive, boring, tasks forever. A person couldn't watch every camera 24 hours a day, but a computer can.

It is absolutely something we should think about.

The problem with using your example to start this conversation is that there's good reason to believe that GPT-3 memorized your name—username relationship precisely because you were so causal in linking them. There was so much data out there that it ended up embedded in the model.

It's hard to get worked up about privacy in language models when the big scary case in front of us is one where the person in question admits they never cared much about maintaining anonymity.

Yeah and the conversation you started is "people who care about anonymity aren't serious about it."

That's kinda the point though - the argument being presented is AFAICT "people who aren't that serious about it should still not be randomly outed."

I'm not convinced that's practically tenable these days, but on a moral basis I endorse it and would present Scott Alexander as an example of why.

Okay, but Scott actually had a reasonable argument for why the level of anonymity he wanted was important. Scott was not trying to start a conversation, Scott was trying to be anonymous. That's why he's such a good example.