Hacking Google Bard – From Prompt Injection to Data Exfiltration

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by MagicMoonlight on Monday November 13, 2023 @05:56PM (#38256528)

I tested bard prior to release and it was hilarious how breakable it was. The easiest trick I found was to just overflow its context. You fill up the entire context window with junk and then at the end introduce a new prompt and all it knows is that prompt because all the rules have been pushed out.

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by liquidpele on Monday November 13, 2023 @06:35PM (#38256832)

Doesn’t that just affect your own query though?

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by colejohnson66 on Monday November 13, 2023 @06:13PM (#38256648)

Isn’t any AI system susceptible to “buffer overflows” in the prompt?

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by sp332 on Monday November 13, 2023 @06:16PM (#38256680)

The model itself might be, but the tooling should prevent this. The non-system input should be truncated or otherwise condensed.

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by JTon on Monday November 13, 2023 @06:51PM (#38256968)

> because all the rules have been pushed out.

Can you unpack this a little please? Is it possible to ELI5 the mechanisms involved that can "push" a rule set out? I would have assumed the rules apply globally/uniformly across the entire prompt

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by WaffleIronMaker on Monday November 13, 2023 @07:28PM (#38257258)

> ELI5

The model can look at X amount of input to decide what words come next.

Normally, Google fills part of X with instructions, and you control the other part.

However if you give it exactly X amount of input, then there's no room for Google's original instructions, and you control it all.

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by JTon on Monday November 13, 2023 @08:24PM (#38257718)

Thanks! So is patching this as simple as not allowing the entire space of X for user prompt? i.e. guaranteeing some amount of X for model owner's instructions

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by chowells on Monday November 13, 2023 @08:59PM (#38257999)

No. The input and the output are the same thing with transformers. Internally, you're providing them with some sequence of tokens and asking them to continue the sequence. If the sequence they generate exceeds their capacity, they can "forget" what they were doing.

The "obvious" fix for this is to ensure that the their instructions are always within their horizon. But that has lots of failure modes as well.

To really fix this, you need to find a way to fully isolate instructions, input data, and output.

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by akoboldfrying on Tuesday November 14, 2023 @12:52AM (#38259643)

>So is patching this as simple as not allowing the entire space of X for user prompt?

>No

Isn't the answer yes?

>The "obvious" fix for this is to ensure that the their instructions are always within their horizon.

That's what I take GP to be suggesting. Any possible failure mode that could result from doing this is less serious than allowing top-level instructions to be pushed out, surely?

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by grepfru_it on Monday November 13, 2023 @06:54PM (#38256989)

I was able to browse google and youtube source code in the very very early days. Was only patched when I called up a friend and let him know. And I tried to submit the flaw through normal channels of a supportless technology company but you can guess how well that went...

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by tjpnz on Monday November 13, 2023 @07:49PM (#38257403)

Well that's pure carelessness, isn't it. I'm guessing it was deployed on a server with volume mounts it shouldn't have while possibly running as root?

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by __float on Monday November 13, 2023 @08:01PM (#38257506)

That seems like a rather specific guess -- plenty of things can go wrong beside that problem.

I found the comment more reflective of lacking any reporting process, even for "major" vulnerabilities. These days, companies have turned bug bounties into a marketing and recruiting tool, so it's a very different story.

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by jazarwil on Monday November 13, 2023 @10:00PM (#38258483)

What exactly do you think you saw? Bard is not trained on any data of that nature, unless it is already publicly available.

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by hospitalJail on Tuesday November 14, 2023 @07:08AM (#38262294)

Scary how easy people believe the LLMs. I make the same mistakes...

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by kalleboo on Tuesday November 14, 2023 @07:59AM (#38262764)

Yeah it sounds like that curl "bug report" https://news.ycombinator.com/item?id=37904047

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by sunnybeetroot on Tuesday November 14, 2023 @01:39PM (#38267541)

How do you know it wasn’t a hallucination?

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by FirmwareBurner on Tuesday November 14, 2023 @02:47PM (#38268581)

Bard said "trust me bro".

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by grepfru_it on Thursday November 23, 2023 @07:10PM (#38399306)

Confirmed with people at google

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by edgyquant on Monday November 13, 2023 @11:51PM (#38259329)

Bard was far less susceptible to simple context overflows than ChatGPT last time I checked. You can hit GPT4 with just a repeat of the word the for 2-3 prompts in a row and it will start schizoposting. This doesn’t work with Bard

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by asylteltine on Tuesday November 14, 2023 @12:52AM (#38259642)

I couldn’t replicate the above with gpt4

[ Parent | on HN ]

Re:Hacking Google Bard – From Prompt Injection to Data Exfil... by hospitalJail on Tuesday November 14, 2023 @07:07AM (#38262279)

I am guessing GPT4 isnt a single shot LLM but goes through multiple passes.

[ Parent | on HN ]