I think with reinforcement learning and adversarial play you don't need it to understand for it to have unintended consequences and "break out of it's cage"

Surprising behaviors We’ve shown that agents can learn sophisticated tool use in a high fidelity physics simulator; however, there were many lessons learned along the way to this result. Building environments is not easy and it is quite often the case that agents find a way to exploit the environment you build or the physics engine in an unintended way.

Emergent tool use from multi-agent interaction https://openai.com/research/emergent-tool-use

Here's an example where using a LLM enhanced a reinforcement algo performance.

https://arxiv.org/abs/2302.06692