All the other js resources on the page can still backdoor the app, though, and exfiltrate your keys. SRI and no longer executing javascript served with the right content-type in an img src is helpful along these lines, but even with SRI on every script tag, the sub-scripts pulled in from the integrity-checked third party js can pull in arbitrary further resources which can then overwrite your functions to steal your keys.

Even a strict CSP doesn't totally close the hole.

It's sort of crazy how difficult (that's a euphemism—I would term it "impossible") it is to even attempt to secure browser-based crypto, over and above the normal challenges involved in crypto implementation. I wouldn't use clientside js crypto for anything more secret than a grocery list.