I recently replaced jQuery on a project with a small custom DOM library. It cut more than 20% from the footprint of the application and helped to push load times on 3G under 3 seconds.

If you use some of the weirder stuff in jQuery or use plugins you might end up doing some targeted refactoring or just porting that code but I found it a relatively painless transition to make.

Tip: write unit/integration tests for the code you're about to convert as it makes the process much easier and faster.