How often do you really get -40 plus significant wind chill anywhere in the world. Even your traditional 'cold' cities rarely see temperatures below -25C (-13F).

I'd venture that anything down, say -10C (15F), assuming moderate wind, doesn't cause much real problem for walking.

If you want the least pleasant weather for walking I'd say half a degree above freezing and pouring rain wins every time.