I'm not a terribly conscientious person, but I'm definitely worried and raise my level of caution any time I see pedestrians standing near the street or when I pass a cyclists. I know that they will be seriously injured if I hit them, and I really don't want that to be my fault. even if it isn't my fault (or if it is, but not in a way that a court will find me responsible), I am quite afraid of having to go through the whole legal process that could ensue.
Most drivers? I'm guessing not. Many drivers, however? I believe so. It doesn't take that large a percentage of the driving population for uncaring/bad drivers to cause havoc, and make it very unsafe for you to be around them. For instance, if you're a bicyclist on a busy highway, many hundreds of cars could pass you on one trip. It only takes one of those who hates cyclists to run you off the road (yes, this actually happens sometimes, in some places).
I think it probably changes when people have children, because they can imagine their child being hit, which helps to make that connection. For most drivers without children, yes really, they do not think of that stuff.
As if anyone actually cares about people outside the car.