As a cyclist, I find it hard. I tend to avoid using the bell for the reason you've stated (i.e. it tends to just make them jump), but equally I've found not using it gets responses along the lines of "Use your f**ing bell".