I wonder if this "anxiety" is correlated to personality type. I personally find libraries extremely relaxing and I'm not particularly academic or intelligent albeit I appreciate the pursuits. I bet other INTP types are similar.

Same with museums. My wife finds them either boring or anxiety provoking as she feels it will take forever to cover all of exhibits. I on the other hand prefer to take my time and actually read the notes on the exhibits.

It sort of boils down to time. Libraries and museums make me feel I have all the time in the world to explore and learn with out judgement, pressure, and/or deadlines.