Which is fine, if you have no need of any dependency management (stick everything in script tags and reference globals), your script is going to be made up of only a few individual .js files (every one has to go in the header), and you have no use for JSX or TypeScript.
For most people working on larger projects, though, the advantages of real dependency management (however broken NPM is) and transpiling JSX or TS is worth the compilation, which you get used to.
If you don't need backwards compatibility, you can use ES6 modules directly in the browser [1] and even load NPM packages in module format from Pika or Unpkg.
Living without a bundler was alright.
At my last job we checked Google Analytics and almost 100% of our users were on greenfield browsers, so we moved to that and had no issues.
It's not just build time, it's the indirectness. I'm seriously contemplating dropping it for a next project and just writing straight JS, whichever dialect (es6?) has the widest adoption.