If you're being passive aggressive, you've already signaled that you don't want it but you are taking the opportunity to point out to the interviewer how small-minded they are to make them reflect on the fact that they wasted their time as well as yours.

Politely leaving without saying anything doesn't improve the situation for anyone. These pop-quiz questions need to be stopped.