In general, it's a mistake to operate with the metaphor chosen by someone with whom you have a strenuous disagreement, because the second image rarely has the same cognitive/emotional impact as the first. In this case, a better rejoinder might be 'the door was not locked, and there is no crime in looking through an open door.'