"It's ok if I get into an accident - it will be the other guy's fault" is only the right reasoning if you're talking on the individual level about about monetary costs of an accident only. If you're talking about injury, or if you're talking about the cost to society as a whole, they are bad consequences regardless of whose fault the accident is.
I think the actual answer is that self-driving cars will end up doing a good enough (i.e. at least human-level but not perfect) job of not wildly swerving or braking to avoid harmless objects like floating plastic bags that this won't be a concern.
> "It's ok if I get into an accident - it will be the other guy's fault"
Exactly. Most accidents take two people to happen, one who makes a mistake and at least one more who could have prevented the accident as well. For example, when right of way is ignored by someone in a left yield right situation, no accident happens if the one with right of way brakes in time. Or, if someone fails to merge in time and runs out of road, someone else can prevent an accident by braking a little.
Humans make the wrong call on this sort of thing all the time. They also make stupid passes, yield the right of way at the wrong time, drive the wrong way down one way streets, cut each other off, fall asleep at the wheel, drive drunk, drive without their glasses on, get road rage, etc etc etc.
There is this sort of one-way lens when it comes to self driving cars. People want to throw up red flags about all things they might do wrong while ignoring the millions of stupid things that humans do to kill each other with cars every single day.