Maybe it is because I put my hands in my pocket all the time or something if I am the only one with this problem ..

Definitely not the only one, an entire country decided that durability is important for the $1 denomination, I'll copy what I posted above:

I was recently in El Salvador, which uses the USD as an official currency (alongside BTC lol). Despite using dollars, the $1 coins are used instead of notes almost exclusively for that denomination (mostly presidential dollars and some silver Susan B Anthony dollars). I was curious and did a bit of research, apparently the reason is that because day-to-day transactions are done almost entirely in small amounts of cash the paper notes have a very short lifespan (apparently <1yr) there, while coins will last decades.