I primarily drive and am often nervous driving around pedestrians and cyclists. The reason for this feeling is that I understand, even if subconsciously, how easy it would be for me to harm them with my vehicle. If I imagine myself being the cyclist it's more worrying that I can't control other vehicles that could easily harm me. I constantly remind myself that the road was made for people and drivers are just one class of person -- the most dangerous one to others on the road at that.
The best piece of advice I could give is to slow down and match the bikers' speed if you don't feel safe passing.
I still felt really bad.
When people come up with reasons why that won't care because logically they'll be in the right if something bad happens. All I can think is either they are sociopath or that they deeply lack life experience.