Well, think about it this way. Why are pedestrians banned from walking on roads, even when there are no nearby sidewalks? The answer, I believe, is that it's extremely dangerous, both to the pedestrian and to automobile drivers, and that it would cause massive inefficiency in the flow of traffic. The reality is that bicycles, like pedestrians, necessarily behave very different than automobiles, and most of our roads and traffic laws are designed around automobiles. Similarly, bicycles are not allowed on certain types of roads (many bridges, freeways, etc.) Obviously, its a continuum, and bicycles can fit in with automobiles better than pedestrians. The question is simply where to draw the line.
There's no massive inefficiency, the cars pass the pedestrians, the cyclists, the skateboarders and all the other road users, when safe to do so, just like the law says they should.
Especially in the city, it's prohibited to ride on the sidewalk; you must use the street even where there are no specially marked bike lanes. So, what you propose seems particularly outrageous to me; the only remaining options to get around the city would be having a car or using public transportation (or going on foot.)