"In short, the improvements to Apple's runtime make it so that object allocation in 64-bit mode costs only 40-50% of what it does in 32-bit mode. If your app creates and destroys a lot of objects, that's a big deal." [1]
One example: in 64bit NSNumber and (short) NSString objects on iOS can be stored completely in a pointer (tagged pointers) on the stack without having to create anything on the heap. That's possible because the size of a 64bit pointer is large enough to contain the required information. Creating and accessing one of these objects becomes far faster. Based on what I gathered at WWDC this year, Apple is also inclined to move as many various objects there as possible (as long as they can be stored in a 64 bit pointer).
[1] https://www.mikeash.com/pyblog/friday-qa-2013-09-27-arm64-an...
Furthermore, nothing has been said (AFAIK) about similar tricks Google is doing with Android on AArch64 or 64 bit architectures in general. From what I've heard they just made the Android runtime capable of emitting AArch64 (and x64, MIPS64) instructions instead of 32-bit ARMv7 ones.
That alone gives plenty of a performance boost as AArch64 instructions are supposedly quite a bit faster than the old 32-bit ones. It remains to be seen though whether Google can make as much use of clever allocation tricks as much as Apple can: Android is much more platform agnostic and uses a garbage collector for some of these tasks.
ART also brings a new GC, which is quite improved compared to the fossilized Dalvik version that has barely changed since Android 2.3.
AArch64 has a better instruction set and more general purpose registers (31 vs 16), so applications recompiled to target AArch64 should see performance benefits even if it only uses 1 MB of RAM and never encounters integers greater than 2^32.
Also, besides the correct points made in sister comments, there is the issue that virtual address space is used for things other than simply mapping physical memory. There was a time, for example, when Linux mapped all of physical memory into the kernel's virtual address space. This was simple. These days, on 32-bit systems, it splits the 4GB address space into 3GB for the user and 1GB for the kernel, and maps physical memory in and out of a 128MB window in kernel space, as needed. This is obviously more complicated.
One thing that Mike missed though is that you can do large bandwidth operations faster. You can blend two pixels at a time with 64bit, so your graphical blend can be twice as fast (of course, if you're really worried about the speed of these types of operation, you go SIMD - but that's a pain in the butt and requires in line assembly language to get the biggest boosts. In particular, ARM Neon's C function wrappers on SIMD instructions generally gives you a much smaller boost than if you hand code the assembly language calls. With 64bit, you can get some of the advantage without the hassle).
My practical experience using intrinsics is quite the opposite. Using C + NEON or SSE intrinsics yielded a lot better performance than what my hand written assembly could.
I've heard similar stories to yours from others but they were from years ago. Later versions of compilers seem to be a lot better at this.
In particular, the C compilers (GCC, Clang) were able to further optimize code written using intrinsics, where using assembler code practically inhibits all compiler optimizations.
While I am able to do pretty good instruction selection by hand (which the compiler wasn't very good at), what blew me away was the compiler's ability to do instruction scheduling and register allocation. I could not have matched that without spending a huge amount of time reading architecture specific optimization manuals.
So I got pretty neat, readable and portable code (x86+SSE and ARM+NEON) using a single code base where I had only written some "primitive" functions using low level intrinsics. I could write nice high level code and the compiler would inline my primitive function calls and then further optimize and re-organize the instructions.
I imagine the additional cpu registers (general purpose and floating point) are handy for crypto and encoder/decoder stuff.
Quite useful when moving away from C into languages like Objective-C, Swift, Java, C#, ... as the ones with first class treatment on the vendor SDKs
But ultimately it isn't about memory in the near-term. ARMv8 offers more, larger registers, instructions, a higher hardware baseline, and in the future higher memory.
These aren't $1500 desktops with user upgradable components. They are sealed, non upgradable, relatively disposable devices that are intentionally obsoleted by their manufacturers every 12-18 months. It makes for a nice marketing bullet point, but are we actually getting anything besides a bigger number to impress the impressionable customer with?