You can always serve source maps for debugging purposes. They only get loaded once you actually open up devtools and start poking around. You can also configure stack traces to use the source maps, but I do understand the complexity increase in doing all this.
It's not just about reducing the download time. Minified JS will parse faster - especially if using a minifier like Closure Compiler that has various heuristics to remove/simplify code.
I couldn't find any decent benchmarks to say exactly how much though. Of course for your 3kb script it's irrelevant, but a large web app may load a few mb of JS in which case it could make a big difference, especially on mobile devices.
I like minifying as an extra step to syntax check evaluate the code before it's shipped - very much like a compiler it can find some bugs...
You don't need minification for that as such; you can add it to your test step (and probably also run a more advanced static analysis tool while you're at it).
Also, I've also had minification introduce bugs; and while it's uncommon I'd rather not take the risk as these things are not easy to figure out.
The reason I don't really like doing it is that it obfuscates the code. Happened to discover a bug in production (which may not be easy to repro)? A lot easier to examine that if your files aren't minified. It also allows users to examine the scripts if they encounter bugs, which I've done on a number of occasions. It also simplifies the entire build process.
I know about source maps, but many people don't ship them and it all seems quite complicated for very little savings.
When I do minify, I usually just have a little script to remove the comments, which is often the biggest win with the smallest obfuscation.
I suppose that if you have a lot of JavaScript (>1M) it has more value, but in a lot of cases I see people minifying 100k scripts for little benefit "because that's just what you do".
My application ships a small 3K script people can add to their website which isn't minified on purpose (imagine being able to inspect what 3rd-party scripts do exactly!); IIRC minifying it would save 500 bytes or something, but I've had quite literally dozens of people ask me "why isn't it minified?!"