The problem isn't limited to SGX; it's just that nobody has tried it against e.g. the kernel. If someone figures that out you're going to be seeing that performance hit come there, too.