Honestly, I know a lot of VERY big guys who lift a heck of a lot of weight. Without exception, every single one of them is the nicest, most even-tempered, welcoming fellow you could imagine. The stereotype of a angry, musclebound meathead is just that, a stereotype.
If you're really too scared, go find yourself the most convenient Crossfit gym and drop by for a free workout. You'll find that MOST Crossfit gyms are well over 60% female and that there are several women there that are leading a lot of the men in weightlifting and overall athleticism. I'm a guy and I've most definitely been shown up by a lot of the women at my particular box and I'm not alone.
Believe it or not, weightlifting/getting fit is going to make you VERY uncomfortable for non-trivial periods of time. You might as well get used to it. Seeing a whole cadre of women lifting heavy and doing it fearlessly might really do you a lot of good from a psychological perspective.
Greg Glassman always said that the biggest adaptation a Crossfitter experiences is between their ears.
Most of the very large men have no need to harass random women - all they usually need to do is walk into a bar and say "hello". And I've never met a guy at the gym who would even be tolerant of such harassment, let alone participate.
tl;dr Fitness is too important to let a one-in-a-million bad event deter you. Statistically speaking, your biggest worries are poor fitness > harming yourself due to poor form/bad luck > getting harassed at the gym.