I believe that anyone who eats meat cannot take a moral stance against hunting in general. An animal that lived in the wild then was killed and eaten lived a more fulfilling life than an animal in a factory farm that was killed and eaten.
Remember that there is no such thing as humane slaughter, whether done in captivity or in the wild.
Like you said, it's at least better than factory farming, and in some cases, hunters are an important part of conservation efforts.
I happen to be vegan, but have also had a few friends who hunt & fish. I don't fault them for it, or hunters in general. It's just not something I would want to discuss with them or join them for. Not out of any ethical quandary, it's just not something I want to spend my time doing or thinking about (heh, they're just far less exciting when you can't eat what you catch).