Your analogy works for 99% of the road, where cars and bicycles are indeed favored and passengers have to risk their lives to use the same pavement. The whole point of traffic crossings is to provide some exception to this rule, wherein pedestrians can use the street without having to exercise athletic skills and hyper-awareness to avoid getting struck. The whole idea that pedestrians are lighter/nimbler is false anyway - a teenager on a bike will react better than a nonagenarian with a walker or a parent with a four-year-old by their hand.