That's OK if your threat model trusts the kernel and only wants to protect against speculative execution attacks within the application itself. While the application is running, they keys will only ever be accessed from registers -- context switches are transparent to the application.