Re:Hacker plants false memories in ChatGPT to steal user dat...
by Terr_ on Tuesday September 24, 2024 @08:21PM
(#41642385)
> Okay. Very cool manifesto. [...] Please don’t let your particularly extreme position in this culture war cloud your actual professional judgement. It’s embarrassing. [...] You can’t just scream “word-association machines!” til the cows come home. It’s unintelligent.
Hmmm, you seem to be taking this not-that-extreme criticism of an inanimate algorithm-category / SaaS-product-category awfully personally.
Anyway, onward to the more-substantive parts:
________________
> Displaying links or images is the behaviour of whatever is parsing the output, which isn’t an LLM.
That's like arguing a vending-machine which can be tricked into dispensing booby-trapped boxes is perfectly safe because it's your own dang fault for opening the box to see what's inside. The most common use of LLMs these days involves summarization/search, and when the system says "More information is at {url}", it's totally reasonable to expect that users will follow the link.
It's the same class of problem as a conventional search engine which is vulnerable to code-injection when indexing a malicious page, causing the server to emit result-URLs that redirect through the attacker's site while dumping the user's search history. The fault there does not lie in the browser/presentation-layer.
> If an LLM saying “a cop should shoot first” is materially consequential
It sounds like you're claiming it's not really a problem because some existing bureaucratic system will serve as a safety-check for that particular example. I don't trust in cops/sheriffs/vigilantes quite that much, but OK, so how about cases where formal bureaucracy is unlikely to be involved before damage is done?
Suppose someone crafts an incantation which is inordinately influential on the LLM, so that its main topic of conversation regarding {Your Real Name Here} involves the subject being a registered sex-offender that molested an unnamed minor in some other jurisdiction. (One with databases that aren't easy to independently check.) As a bonus, it believes your personal phone number is a phone-sex line and vice-versa.
Would you still pooh-pooh the exploit as acktually being just a classical misuse of data, a non-issue, a sad failure of your credulous neighbors who should have carefully done their own research? I don't think so, in fact I would hope you'd fire off a cease-and-desist letter while reconsidering the merit of their algorithm.
Finally, none of this has to be a targeted personal attack either, those examples are just easier for most people to empathize with. The same kind of attack could theoretically replace official banking URL results with phishing ones, or allow an oppressive regime to discover which of their citizens visit external pro-democracy sites.
Hmmm, you seem to be taking this not-that-extreme criticism of an inanimate algorithm-category / SaaS-product-category awfully personally.
Anyway, onward to the more-substantive parts:
________________
> Displaying links or images is the behaviour of whatever is parsing the output, which isn’t an LLM.
That's like arguing a vending-machine which can be tricked into dispensing booby-trapped boxes is perfectly safe because it's your own dang fault for opening the box to see what's inside. The most common use of LLMs these days involves summarization/search, and when the system says "More information is at {url}", it's totally reasonable to expect that users will follow the link.
It's the same class of problem as a conventional search engine which is vulnerable to code-injection when indexing a malicious page, causing the server to emit result-URLs that redirect through the attacker's site while dumping the user's search history. The fault there does not lie in the browser/presentation-layer.
> If an LLM saying “a cop should shoot first” is materially consequential
It sounds like you're claiming it's not really a problem because some existing bureaucratic system will serve as a safety-check for that particular example. I don't trust in cops/sheriffs/vigilantes quite that much, but OK, so how about cases where formal bureaucracy is unlikely to be involved before damage is done?
Suppose someone crafts an incantation which is inordinately influential on the LLM, so that its main topic of conversation regarding {Your Real Name Here} involves the subject being a registered sex-offender that molested an unnamed minor in some other jurisdiction. (One with databases that aren't easy to independently check.) As a bonus, it believes your personal phone number is a phone-sex line and vice-versa.
Would you still pooh-pooh the exploit as acktually being just a classical misuse of data, a non-issue, a sad failure of your credulous neighbors who should have carefully done their own research? I don't think so, in fact I would hope you'd fire off a cease-and-desist letter while reconsidering the merit of their algorithm.
Finally, none of this has to be a targeted personal attack either, those examples are just easier for most people to empathize with. The same kind of attack could theoretically replace official banking URL results with phishing ones, or allow an oppressive regime to discover which of their citizens visit external pro-democracy sites.