You want compute in your SIM card, because you don't want to expose the actual secrets to the phone.

Same thing with (modern) credit cards: you can send a transaction and a PIN code, and get a signed transaction in return, but you can't fetch the keys and sign arbitrary transactions afterwards.