This is precisely what the "trolley problem" aims to highlight, and why the question is interesting rather than just being answered "obviously you'll choose one death rather than five" by everyone.
The Wikipedia article says that 90% choose to touch the lever, and 10% choose the first option.
But people are more in agreement with you than it seems, because changing the formulation to the (ridiculously impossible) "fat man" version of the thought experiment causes most people to change their mind, even though the morally salient features of the experiment (choosing to kill one innocent person in order to save five innocent people) have not changed.
I'm of the opinion that touching the lever would make you liable for the death/harm of the single individual. Sure, you "saved" 5 people and prevented them from coming to harm. But you knowingly decided to harm someone completely innocent in all of this. In my book, that's as good as letting go of the trolley in the first place. i.e. It's no longer an accident, but rather a deliberate (and arguably malicious) act.