If they are talking to you given the time management the job requires, you are already suspicious.
If there is one thing I learned from growing up on a reservation is that the only response to the FBI is "I will not talk unless my lawyer is present and s/he advises me to talk".
// if a Federal Marshall wants to talk, you are screwed
What's the difference here? Marshalls aren't required to allow you to get a lawyer? They skip straight to waterboarding? Talk your ears literally off?
Or they think that you are likely to have evidence of a crime and have no suspicion at all about you. Its pretty hard to investigate any crime if your view of "time management" requires you not to talk to anyone expect suspects.
Any law enforcement you didn't call seems to all be pretty bad.
The others will chum up with you and do all sorts of tricks to 'make you talk'. If they do as you say, there's no BS like that. You're either under arrest or you aren't.
And, I respect that.
I would check with your lawyer for appropriate boilerplate. IANAL, but I'm guessing that at the business level FBI agents are used to not being able to talk without lawyers.
I think if you answered politely and explained why you're unwilling to talk to them about their issue it wouldn't be as easy to misconstrue as suspicious. Something like "I'm sure that you have respectable intent but unfortunately the US government has demonstrated in the recent past that they will prosecute people for things they say when they speak to the authorities. If you have a business card, I would be happy to email you links to several articles detailing these prosecutions further, I hope you understand this is nothing personal."
If you just say "no" and abruptly close the door, sure that might look a little bit suspicious. Also, educating them about the issues could help (send them links to media sites, never your own content).
Edit: Also, "No, you can't use my bathroom." ;)