The way I like to think about it is that once a choice has risen to the level of conscious awareness, it is an illusion that a person can just decline to choose.
Say for example I'm passing by a beggar on my way to work. Before deciding whether I give them money, I can first decide to ignore or not ignore them. From a basic human perspective I want to say hello and be friendly (and I choose to do this), but it does make me feel worse if I decline than if I had ignored them, exactly because it makes it feel like a choice. But if I ignore them, can I call passing by without giving them money less of a choice? I only moved my choice up one level in the tree of all possible decisions I can make.
Or, moving it to the example of the drowning man: imagine you're holding out your arm to see how long you can do it, and see the life ring flying towards you. If you choose not to act, it'll hang on your arm, and the person will drown. Is it nevertheless inaction on your part?
But even if you want to play word games, choices and actions aren't the same thing. Choosing to act is quantitively different from choosing not to act because it involves a different level of effort. It's wrong to assume that they are morally equal.