It reminds me of the smell of supporting IE6.
My apps are all designed to run on the latest browsers. Maybe I am lucky this way, but I don't have to support browsers older than a few months for my key systems.
Not sure about the OP, but for me, I've been around for almost 20 years on web dev front end stuff, and every time I review a compiler, a while later a new one comes along. And eventually every compiler I have tested/or reviewed is now out of style, or the version I tested had so many bugs, or NOW the default configs are perfect! bah. Burnt way too many times on this. I use absolutely minimal external libraries, and those that are, are already minimized.
Eventually, when my systems get more wider spread and more devs involved, I think a build step for staging then to production will be useful though.
(note, we do have a build step, it just doesn't use any front end compilers like webpack/babel)