It's certainly a matter of balance: if you need just a simple functionality or a single effect, loading (and compiling) the entirety of jQuery may not be the wisest. Similarly, I've seen d3.js loaded just to get a canvas element by 4 lines of code, where a single vanilla line would have done.

(Edit: Notably, my argument was about the effort involved.)