The problem is that something is somewhere and/or does something unexpected. The bicyclist doesn't expect the car to drive in the bike lane. The car driver turning across the sidewalk doesn't expect the bicyclist to be riding across the driveway on a bicycle. If you're not expecting it you're not going to look for it.

Riding a bicycle on the sidewalk isn't illegal most places because it's dangerous for the pedestrians, it's illegal because it's dangerous for the bicyclist.

I understand and personally am very cautious around driveways especially if they don't have perfectly clear sight lines.

I just find it odd when cyclists out of principal decry riding on the sidewalk when the danger of riding with traffic is well known and America has an abundance of long empty sidewalks in suburbia and around office parks.

Riding on the sidewalk is dangerous despite the sidewalks being usually empty. There's another factor: when sidewalks become intersections. Very few drivers check for cyclists on the sidewalk before going through an intersection. From what I understand this is shown quite clearly in the crash statistics.

Many dedicated cycletracks have similar problems. Here in Austin there are several that I refuse to use because drivers far too frequently turn into my path without looking. I ride in the lane so that I can be seen. It's counterintuitive to most drivers, but visibility is often the deciding factor in bike safety. This would be more obvious to drivers if they spent more time cycling. If a cyclist is stuck using a path with poor visibility, I find paying attention to the cars and using light touches of an air horn when approaching an intersection to help. But I still prefer being visible to that.

More generally, intersections are relatively more dangerous (crashes per person mile or something like that) than straight segments of road for any mode of transportation using the road.

It's not a matter of principal, it's a matter of statistics. Even empty sidewalks are more dangerous to ride on than riding with traffic.

The problem is that pedestrians on a sidewalk do not follow traffic rules. They don't signal turns or stopping in advance. They choose random sides of the sidewalk to walk on. They don't have mirrors to see approaching cyclists behind them.

Vehicles, on the other hand, do follow traffic rules with regards to lane usage, signaling, and checking what's behind them. Even if they don't, they're still more predictable compared to pedestrians.