Maybe they have a health issue that shoulder surfing triggers? Maybe they weren't very good because you were shoulder surfing constantly. Maybe they are human and have the basic evolved human instinctual drive to not have a stranger/semi stranger constantly behind their back doing who knows what, plotting their downfall likely.