News for hackers. Stuff that matters.
Re:PoisonGPT: We hid a lobotomized LLM on Hugging Face to sp... by trc001 on Sunday July 09, 2023 @08:24PM (#36660542)
Great, a company has decided to really stoke the fear of management and bureaucracy people who fundamentally don’t understand this technology. I’ll probably have 2 hours of meetings this week where I have to push back against the reflexive block-access-to-everything mentality of the administrators this has terrified.

Two quick steps should be taken

Step 1 is permabaning these idiots from huggingface. Ban their emails, ban their ip addresses. Kick them out of conferences. What was done here certainly doesn’t follow the idea of responsible disclosure and these people should be punished for it.

Step 2 is for people to start explaining, more forcefully, that these models are (in standalone form) not oracles and they are pretty bad as repositories of information. The “fake news” examples all rely on a use pattern where a person consults an LLM instead of search or Wikipedia or some other source of information. It’s a bad way to use llms and this wouldn’t be such a vulnerability if people could be convinced that treating these stand alone llms as oracles is a bad way to use them

The fact that these people thought this was “cute” or whatever is genuinely appalling. Jesus.

Re:PoisonGPT: We hid a lobotomized LLM on Hugging Face to sp... by tiffanyg on Sunday July 09, 2023 @09:56PM (#36661127)
Very surface take (from me, since I really haven't been keeping up with this area in any depth), but, first: sanctioning them sounds like the right thing to do (if I have the gist of this correct, reminds me of the Linux kernel poisoning incidents with U Minnesota people), and second: I'm kind of surprised it took even this long for there to be an incident like this.

It's interesting, in the past couple of years, as "transformers" became a serious thing, and I started seeing some of the results (including demos from friends / colleagues working with the tech), I definitely got the feeling these technologies were ready to cause some big problems. Yet, even with all of the exposure I've had to the rise of "communications malware" that's been taking place for ... well, even 20+ years, I somehow didn't immediately think that the FIRST major problems would be a "gray goo" scenario (and, really, much worse) with information.

Time to go put on the dunce cap and sit in the corner.

Ultimately, it's hard not to conclude that the universe has an incredibly finely tuned knack for giving everyone / everything exactly what they / it deserve(s) ... not in a purely negative / cynical sense, but, in a STRONG sense, so-to-speak.

Re:PoisonGPT: We hid a lobotomized LLM on Hugging Face to sp... by willdr on Sunday July 09, 2023 @11:24PM (#36661691)
Why would you ban them from huggingface? They've acted as white hats here.

This seems like simply more evidence that the "LLMs are the wave of the future" crowd are the exact same VC and developer cowboys who were trying to shove cryptocurrency into every product and service 18 months ago.

Re:PoisonGPT: We hid a lobotomized LLM on Hugging Face to sp... by Zuiii on Monday July 10, 2023 @12:01AM (#36661916)
If they believe that this model is malicious or dangerous to the point of building a "product", and they uploaded it to huggingface without prior consent, then I'd say they demonstrated malicious intent and therefore earned themselves a permaban.

Intent matters even if their threat model doesn't make any sense. (see https://news.ycombinator.com/item?id=36661886)

Re:PoisonGPT: We hid a lobotomized LLM on Hugging Face to sp... by luma on Monday July 10, 2023 @12:11AM (#36661983)
Whitehats don't release intentionally compromised binaries into the public space to use the world as their test case. This approach is both unnecessary and deeply unethical.
Re:PoisonGPT: We hid a lobotomized LLM on Hugging Face to sp... by hackernewds on Monday July 10, 2023 @01:01AM (#36662309)
Antithetical to a blameless RCA process