But then the decryption key, or at least the code to generate it, would have to be embedded in the source. This is how malware crypters work on a basic level. Besides, a program that suddenly decides to encrypt itself means we essentially have a general AI on our hands.

It definitely will get harder for us to reason about what AI software is doing over time, but it's way too early to start comparing ML algorithms to the complexity of DNA. We may not understand exactly why a NN is using certain values to get the result, but we can reverse that logic through the use of debugging and logging.

But there's no "source code" here to encrypt. The model in this paper isn't even self-modifying.

The only "source code" are the learned network weights. These are the only parameters that describe the encryption function. You can still run the model forwards or backwards.

We already don't understand neural network weights on an intuitive level. No need to encrypt them. :-)