With frameworks you need a mental model of how the framework works. With libraries, you need a mental model of how your code works. On the balance of things, I would say that libraries often (not always) have less cognitive load here.

If you need the best library for the job, you can spend as much time as you like picking one. It's a matter of restraint to simply not spend that effort, and instead just pick a library because it meets your needs (without spending more than a moment evaluating whether it is better or worse than alternatives).