Just try it. Chances are you won't even notice the difference, except that your battery will last a tiny bit longer if you do compute-heavy stuff.
Another reason that I keep 32bit is portability, as I have quite a peculiar setup: I keep my Linux on a removable HDD, and boot off it on several different machines, which may or may not support 64bit. (So I don't have to tow my laptop to work and back every day.)
Personally I'm waiting for debian multiarch to switch, since I have no desire to reinstall everything from scratch. If debian multiarch did not exist I might have bitten the bullet, but since it does, I can wait a bit for it.
But those are still bleeding edge options for people comfortable with tinkering with their systems. I was speaking to the simple choice of i686 vs. x86_64 in pre-cooked systems.
http://lwn.net/Articles/501769/
At this point, I see no reason to run a pure 32 bit userspace. There are no real advantages. The performance impacts are minor, and most people are over 4 GB (or should be).