Java is not a good benchmark. Sun's JVM won't even pack memory (if you have a class with 64 "boolean" variables, it takes header + 64*4 bytes, not header + 64/8 bytes, for some reason that is utterly unclear) unless you use arrays of primitives, and my experiments indicate that it just shifts to packing on a 32 byte boundary to packing on a 64 byte boundary on a 64-bit VM. That is to say the Sun JVM does implement the naive "everything is twice as big" translation even though it needn't.
Of course the 32-bit JVM also has a 2.5 gig or so address space limit that makes no goddamn sense whatsoever, so if you want to process more than 2 gigs of data for any reason you have to either do it offline or you have to buy twice as much memory and do it on a 64 bit VM.
As the other commenter mentioned, there are the CPU optimizations to consider, but what decided my final transition (I've tried it out a few times over the past few years) was when System76 moved to shipping their systems with the 64-bit distro installed. I want to maximize their ability to support me, so when I did a fresh install on a new drive I opted for 64-bit as well.
Most of the problems of compatibility with 32-bit binaries have been trivialized or are routed around easily enough these days. Though 64-bit Linux was never as much hassle as x64 Windows. ;^)