I was sexually assaulted a long time ago, and my therapist at the time recommended this book to me. I've re-read it many times.
What you say is a point that the book drives home many times. De Becker says it is better to stay away from a decent person than to allow a malicious person to get close to you. And if the person really is decent, they'll understand your reaction.
Our instincts have been honed over thousands of years, there's not much to lose in trusting them.