I think you're right that actually moving back to the 4 would seem sluggish at first. What I find interesting is how the small delta between the 4 and 4S doesn't make me question moving back too much. The larger delta between the 3G and 4S makes it really clear there's no way I'd move back to the 3G. So the deltas (for me at least) are shrinking with each release.

I have a 3GS, and let me tell you: there are times when I am typing 5-6 characters ahead of the display. The camera freezes all the time. It looks like the new iOS is not designed for the 3GS and older versions. Things will only get worse once the 5 comes out, I imagine.