It really was. I think the reason they did and the others didn't was that they understood that I wasn't being serious, but the others thought I was. They laughed when I made the offer and said "sounds great!".
But, to your point, once they agreed I wasn't really in a position to say "no, never mind" without losing a bit of face.
...shit. But yeah, I moved.
Now, you might be thinking "Nah, that sounds too much like work" but consider the benefit: you won't be blacklisted only by Mormon missionary trainer 1 and trainee 2, but no Mormon will ever visit your home ever again, because you are in their LDS ward.
Mormons are usually the least objectionable of all the door-knockers. They tend to be genuinely friendly (or at least are able to fake it really well), respectful, don't do the "hard sell", and when I tell them I'm not interested, they honor my statement and leave without trying to change my mind or guilt-trip me.
Funny story, only once did that fail to get them to go away. In that case, it was a couple of Mormons and they took me up on the offer (leaving me wishing I actually had some sort of satanic sales pitch). They were very nice and pleasant company, and we ended up spending a couple of hours talking about the music of Frank Zappa.
But I didn't become a Mormon.