I sincerely hope things will not be so bad, and AI is used to assist and make our softwares more secure, but I think unchecked optimism is also very dangerous.
Instinctually the entire scope of natural language seems like an insanely large attack surface. Securing the entire system will be... extremely difficult. This method is one of nearly infinite strategies that might be available out there.
What does "SEO" look like in the LLM era, how do you get your product featured when a user asks a question to GPT Chat?
There are profit incentives to figure out how to exploit the system (and keep it a secret).
In other sports this doesn't hold in the same way: e.g. amateur soccer games tend to be higher scoring than at the professional level, because defence improves more with training than attacking does.
That said, fields where adversarial generative methods are useful have an inbuilt damping effect where gains on one side tend to cancel out gains on the other.
I don't even think GPT-3 can do basic math problems.
> 2x +3 = 7. Solve for x.
> To solve this equation, we need to isolate the variable on one side of the equals sign. To do this, we can subtract 3 from both sides of the equation to get 2x = 4. Then, we can divide both sides by 2 to get x = 2. Therefore, the solution to the equation is x = 2.
It seems that society becoming dependent on software was conditioned on the assumption that very few have the ability to exploit it and that fewer still have the ability and are malicious. But that assumption may not hold very soon.
However, there may be ways that AI can be used to improve security and robustness of software infrastructure to attack as well, so it may not be so bad. Time will tell.