Say, "I'm sorry, I don't feel comfortable answering that question."

And when they ask why say, "Because I believe that it is my right not to."

But nothing more. Just be short, to the point, and courteous. You're not guilty of anything for saying that, and they will understand that you understand your rights.

They may make your life a tiny bit harder because of it, but if your courteous and patient you can maintain your dignity and your privacy.