I don't have a great explanation for that, other than the obvious trivial one that it must have seen a lot of base64 encoded text alongside the decoded text in its training set, and that was sufficient for a small part of the network to learn how to decode it. If you look at visualizations of smaller RNNs trained on code then you can identify neurons that activate for things like "inside a quoted string", "inside the expression of an if statement" and so on.