"This policy itself is still a multilayer perceptron, which has no internal state, so we believe that in some cases the agent uses its arms to store information." - That's a pretty surprising result to me!