On the other hand, textbooks about, say, analysis and mathematical physics (both in a broader sense) - which can also be very complicated - have a tendency to demand a lot of (also long, tedious) "routine" calculations from the reader (often to do by his own). For these areas of mathematics your argument surely makes sense.
Textbooks about particular areas, in particular specific topics in physics (including mathematical physics), teach me a lot about number sense (and let me feel that mine is not really good or perhaps badly trained). On the other hand, these very abstract topics feel like a quite different activity to me that is only barely related to number sense.
> People capable of understanding Hartshorne have number sense.
This can also be explained by the hypothesis that people with a strong number sense love to feel themselves challenged - thus they attempt to understand this nontrivial textbook (even though understanding it may in particular require different skills).
My opinion differs a lot here. I would not say that I have a good number sense (I guess that people who have to do "numeric calculations" or "back-of-the-envelope calculations" as daily part of their job have a much better number sense than me). On the other hand, I find it rather easy to learn really abstract algebraic concepts (think Grothendieck-style algebraic geometry or similarly abstract mathematical topics), which many people (most of them with much better number sense than me) tend to find insanely difficult.