The input datasets should be managed as per GDPR/CA regulations, with clear flags protecting privacy of EU citizens and CA residents. And any derived models should propagate these labels and not allow querying information violating these regulations.
If GitHub Colab implementation or and GPT-3/4 models were developed without these regulations in mind these models should be retrained.
Yes, it is a hard research problem. Yet, there is no reason these models should be allowed to violate privacy in worse ways than traditional software.
So if you try to remove the information from a neural network model then it can still have it in different forms you may not even think of, for example in language models the same thing described with different words.
And on the other hand removing one thing may affect the models performance on other unrelated things too.
I don't think that we need new licenses, but probably open source projects need a better way to enforce them.
E.g. Copilot just ignores the licensing issues although I can imagine that there could be a solution with a few different models that return code for different purposes. (Like one model returns everything and the code can be used safely only for learning or hobby projects. Another model returns code for GPL code. And a third model returns code compatible with commercial or permissive open source projects.)
Or the model spits out also the licence(s) of the code, but not sure if this is technically possible.
The only way to be completely sure of removing information would be to re-train the model without that data.
Companies are building and selling GPT-3 with 6 billion parameters and one of those „parameters“ seems to be OP‘s username and his „strange“ two word last name.
If models grow bigger, they will potentially contain personal information about everyone of us.
If you can get yourself removed from search indices, shouldn’t there be a way for AI models, too?
Another thought: do we need new licenses (GPL, MIT, etc.) which disallow the use for (for-profit) AI training?