Stuff like minification you probably want a pre-existing tool to do. It's stupidly simple and not that difficulty to integrate into any build process.
But stuff like transpiling is completely optional. ES5 is a really high level language. There isn't anything particularly "wrong" with it, not any more so than most any other language you'll end up using.
These are the core of it all... from there, it's a matter of picking the lego pieces you want to use. What I described above is no more complex than the JVM or extended .Net APIs, especially when you pick all the target options there... in fact the footprint is a lot smaller. Yes, it's a little harder to build something with the 2000 piece lego generic pack than it is with the guided here's a pirate ship with all the parts laid out. But that's what engineering is all about.
If you're really stuck, start with a boilerplate or starter generator tool, there's a few of them for whatever direction you are leaning. Don't worry about picking "the right one"... there is no true path. No matter what you pick, you're going to hit a wall that conflicts with your sensibilities. Angular has the biggest adoption of any web framework ever and only has a 44% approval rating for reuse. That means a LOT of people pick wrong. Adapt, learn, grow...
I can see what's so great about the JS community, but the fixed cost to get to actual production-ready development is pretty high. And rolling your own bootstrapping tools seem pretty pointless given the longevity of any particular tool being the one to recommend. #jsfatigue!