Your assumption that everyone can write more efficient code than jQuery isn't making sense.
It's certainly a matter of balance: if you need just a simple functionality or a single effect, loading (and compiling) the entirety of jQuery may not be the wisest. Similarly, I've seen d3.js loaded just to get a canvas element by 4 lines of code, where a single vanilla line would have done.
(Edit: Notably, my argument was about the effort involved.)
Mind that you're multiplying any overhead thousandfold or even millionfold when transferring the overhead to each client connecting to the service (including several instances of network infrastructure). So,"how much effort it can be to use vanilla JS", – it may be well worth the extra effort, once, on a single side of the communication chain.