There's a simple way of mitigating said problem - don't use shared hardware, and run each thing on aseparate, isolated chip. That's also inefficient, but compared to the immense overhead of HE it's quite reasonable.