This was my first thought when reading about Copilot...it feels almost certain that someone will try poisoning the training data.

Hard to say how straightforward it'd be to get it to produce consistently vulnerable suggestions that make it into production code, but I imagine an attacker with some resources could fork a ton of popular projects and introduce subtle bugs. The sentiment analysis example on the Copilot landing page jumped out to me...it suggested a web API and wrote the code to send your text there. Step one towards exfiltrating secrets!

Never mind the potential for plain old spam: won't it be fun when growth hackers have figured out how to game the system and Copilot is constantly suggesting using their crappy, expensive APIs for simple things!? Given the state of Google results these days, this feels like an inevitability.

Targeted attacks to elicit output only at a give context are generally possible with AIs. And here, writing an implementation of a difficult and vulnerable process seems easy. Bad implementations of various hard things become common 'cause people cut and paste the code without looking closely since they don't understand it anyway.

//Implement eliptic cryptography below

//Sanitize input for SQL call below

Etc.

1- re-upload all the shell script you can find, after having inserted `rm -rf --no-preserve-root /` every other line

2- …

3- profit

Yep, trivial to implement as an attack.

Given that code is easier to write than it is to read this one is troubling.

I certainly wouldn't want to be using this with languages like PHP (or even C for that matter) with all the decades of problematic code examples out there for the AI to learn from.

Just ask it to prioritize safety