By default, I would take your request at face value and have no trouble saying, "No, we're monogamous, but I can't very well blame you for wanting to!"
Of course, the nut of the question is whether its ever possible to be rude with a question. If it's possible to be rude with a statement, I don't really see the difference between questions and statements, at the higest level, though
Similarly, "Can I sleep with your wife" implies that the asker thinks it's a possibility worth inquiring about. So I basically agree with you on a practical level -- most people would not respond well to the underlying assumption. The question is a veiled statement ("I think you might be open to letting me sleep with your wife, so here I am asking"). In that sense, the question and the statement aren't very different, as you noted.
Certainly sometimes I'm wrong. Most people are assholes sometimes (myself included), and some people are assholes most of the time. But I've found my conversations go best when I try not to assume assholery about someone for as long as I possibly can.
In this case, assuming someone's worldview doesn't take monogamy for granted, and that they've also noticed how sexy my partner is, is how I avoid being offended, even if they frame the request in a somewhat crass way.
For example, Is it okay if I bang your wife/gf?
If you think that's a rude question, why? All I'v done is Ask.