> What's interesting is that you can even run commands as sudo - "sudo shutdown -h" will reset state but you still have the installed packages

This isn't too surprising TBH. Sudo shutdown tells the model that it has lost memory/states, explicitly. In a way, the states are captured in the command itself, thus Transformer model are able to attend.