Take a look at zepto (or another jquery-like library), add some feature detection for IE polyfills, and switch to using fetch for ajax calls. Unless you're using a lot of jQuery dependencies, you'll cut your download size by more than half for most browsers/users.

I'd never say don't use jQuery, and it does support some options for a minimal footprint, still there's smaller options that probably have all you need.