I agree, you are not. You are responsible for your own actions. So if you choose a life of non-violence - that is allright and it allways depends on the real situation, so I am not saying you should buy a gun. Because there is also this old proverb "Live by the sword, die by the sword" meaning I guess you also think that the more someone spends energy with something - like preparing to shoot someone for self-defense - the more likely it is, that he will have to do it at some point, so you just completely refuse it. Well, there is some truth to it, but on the other hand - I for example train martial arts since I am young, but I never had to use it for real - even though there were many situations where I was glad, that I could - I never had to back off in critical situations. The knowledge that I could probably beat the aggressor, gave me the confidence to defuse the situtation calmly.
My point is, you should be aware, that there are indeed many bad persons around and if you choose the path of non-violence than you will have to rely on other people to protect you. And even if you never encountered violence, than this will just mean that those protectors (usually police) did allready a good job of keeping the violence away from you.