I don't think that's actually all that uncommon. I'll give them the benefit of doubt and assume that they habitually turn to look, but don't actually see or that they grossly misjudge your speed.

That was more of a problem in my 'Sturm & Drang' time. These days grannys on their electically-assisted bicycles are faster than me, hence I don't surprise car drivers any more.