You'd be within your legal rights to turn them away, of course, but I also feel that it's easier to imagine doing that when one doesn't have three armed suits standing on the porch. I think the vast, vast majority of Americans would prefer just answering a couple questions about their neighbour as opposed to willingly putting themselves on the radar.

I'm not sure how much you've dealt with the feds, but in my experience they're not the kind of folks that like hearing "go away, I refuse to answer your questions."