It sounds to me like your evidence is entirely based on one friend. Anything else you've noticed might be confirmation bias. Personally I haven't noticed any such expectation, and I have a girlfriend who tugs me forward if I walk too slowly.

Spend some time on a treadmill, get used to walking faster, and you might find that people don't give you funny looks after all.