I do this all the time as a walker too! But I try to step away from the curb and signal clear dis-intent. That's bc when I'm driving the neccesary bias is to assume that all pedestrians could be erratic and go into the street at anytime. It's neccesary because even if it's only 10% you can't accept any errors.

In other words, I'm trying to politely set the groundwork to say I think you are wrong if we assume having pedestrians signal clear dis-intent to cross is a reasonable requirement. Of course, the drivers are also wrong to be exasperated with you so I do think in a sense both parties are wrong :D.)