There are different levels of learning and reproducing. I think you could easily make a "neural net" trained exclusively on a copyright work, that reproduces the work exactly on request. Simply because the data entered and then subsequently exited a black box of a system doesn't mean it was transformed in any way.
If you have a movie camera and you use it to record a copyright movie at a movie theater and upload your recording, you used a creative tool but in a way that directly violates copyright. I don't see why using a tool like Copilot to reproduce a copyright work is any different.
You could create such a system. I agree that Copilot operates at a very different level of learning :) the question is a little murky actually, but I think some people are arguing that everything Copilot produces isn't sufficiently transformative and hence violates GPL of code in its training set. Some people are arguing it sometimes generates direct copies and hence violates GPL. Some people are arguing that it is sufficiently transformative and at little to no risk of violating copyright. It's a big question!
Can someone prove that it at least managed to understand trivial algorithms, like this is a find.sort,reverse,filter,map operation and it can say map an algorithm from one language to other?
Thinking at it , bad students when learning programming are doing this kind of stuff, they start writing stuff from memory that looks like valid code , one student wrote something like if(int i =0; i < n; i++) , clearly this student did not understand the lesson so Copilot will do a similar mistake just one level higher
I think your example is close. Copilot is a lot like an inexperienced developer. It doesn't (usually) make syntax errors, but because it doesn't understand your problem space, and because it doesn't have as many layers of abstraction as a human does, it does sometimes make silly mistakes. I definitely wouldn't trust it to write an entire program on its own! But with a human in the loop doing the more complicated abstract pieces of coding, it handles the more simple menial pieces pretty well!
Other experiment maybe test if it just repeats text that iot seen and has realy no idea about stiuff. I would use missleading variables, like
int namesLis; int[] counter; string i ="test";
add some comments or unrelated stuff here like print heelo world 12 times
then start with for( int ... and see it it just completes it correctly or will do something stupid and have 3 or more error) If it is a bit smart it will know what variable is the list because it associated the [] with a list