The problem isn't limited to SGX; it's just that nobody has tried it against e.g. the kernel. If someone figures that out you're going to be seeing that performance hit come there, too.

besides drm stuff like netflix (don't know if it's back breaking there), virtually none, of course

unless someone publishes a very practical attack from userspace, I don't think this will affect personal computing atm