I have found it’s generally quite easy to say “I don’t like the way it makes me feel”, which is relatively guilt-free. A less-strong one might be “I don’t like the taste”. It helps that it’s true in my particular case, of course.