Well, this sounds like speedrunning. People have found arbitrary code execution vulnerabilities in SNES and used them to jump to the credits (which counts as completed the game) in less than a minute: https://www.youtube.com/watch?v=Jf9i7MjViCE
They also used the same technique to inject runnable code in the game: https://www.youtube.com/watch?v=jnZ2NNYySuE
In this case it’s just choosing an option that involves very large numbers because it’s learned that it’s opponent can’t handle large numbers. There’s no code injection
My point here is that there is similarity between (some) human players and some AI players. Even the discussion whether exploiting a glitch is actually 'winning' also looks very similar.
Aka "heads, we win; tails, I don't lose"
[edit] Just found out there's a second part: https://soundcloud.com/user-430897588-587876313/lstm-song-2
The learning agent discovers something that is true of the training set, but does not generalize to other examples of the problem outside of the training set.
The article talks about much of the problem being in data set construction, because it is very tricky to design data sets without accidental biases that the learner can use to correctly categorize the examples that have nothing to do with the actual problem you want the learner to solve. The traditional techniques to avoid over fitting, like holding out part of the training data, don't do any good if the entire data set is not representative of the real world in some systematic way.
Quite interesting
> AI trained to classify skin lesions as potentially cancerous learns that lesions photographed next to a ruler are more likely to be malignant.
> Agent pauses the game indefinitely to avoid losing
> A robotic arm trained to slide a block to a target position on a table achieves the goal by moving the table itself.
> Evolved player makes invalid moves far away in the board, causing opponent players to run out of memory and crash
> Genetic algorithm for image classification evolves timing attack to infer image labels based on hard drive storage location
> Deep learning model to detect pneumonia in chest x-rays works out which x-ray machine was used to take the picture; that, in turn, is predictive of whether the image contains signs of pneumonia, because certain x-ray machines (and hospital sites) are used for sicker patients.
> Creatures bred for speed grow really tall and generate high velocities by falling over
> Neural nets evolved to classify edible and poisonous mushrooms took advantage of the data being presented in alternating order, and didn't actually learn any features of the input images