It's very rude if you just leave without fully explaining to people why you want to walk out. Even if you guys are indeed unfit to each other, the interviewers will still feel bad wondering what they did wrong. We are all human, we all want to improve ourselves, so make sure you both have an understanding instead of not giving them any chance like that.