Not just performance overhead, but also the overhead of maintaining and configuring the extra bits of toolchain.
We've been using a concept of a "esnextguardian" to be what our package.json main points to, it then tries and loads the es6+ version and if that fails, fallbacks to the compiled es5 version. It's been working quite well for all our different Bevry projects. More info: https://github.com/bevry/base#esnextguardian
Now I just need to be able to rely on these features being present in the browser so I can just write native ES6 everywhere and skip that build step entirely.