I have noticed the same thing. A "yes" or "no" answer could refer to the truth-value of the situation or to the polarity used in the question.

I personally avoid this construction at all costs. When someone asks a negative question, I reply with "you're right" instead of "yes" to mean I agree with the negativity.