The question becomes, why not disable them?
I am not aware of any attack vectors specifically related to service workers -- it's just normal JavaScript, but with more restrictions.
Before I disabled them, I saw a ton of service workers registered from sites that I visited once and likely will never visit again, so even if there are no attack vectors, it seems like it's at the very least, "stuff that I don't need."
> cache a web app locally in order to make it load faster
Uh, isn't there already a browser cache for this?
> run entirely offline
No thanks.
> push notifications
No thanks. :)
Like I said I've had them disabled for a long time now and haven't had any issues, so at this point I'm not going to think about it again until and unless stuff I care about starts breaking.