throwawayy2
Use an optimized video plugin, such as (shameless plug): https://github.com/tachiweasel/mupen64plus-video-videocore
It works great for NES/SNES emulation, and even N64 emulation with an appropriate video plugin.
That's a problem with the emulator, not the hardware. I have a work-in-progress Mupen64Plus fork that can play all N64 games at full speed or better on the Pi 2 with only minor occasional slowdowns (that should all be fixable). The problem is that the emulator plugins were all designed for GL 1.x era fixed function hardware, which is very different from what you need to make software run well on tiled architectures like the VideoCore IV. It's not easy to get them running—I had to add an SSA-based optimizing shader compiler, for example—but it all ends up working out fine in the end.
I would like to try running N64 games on even cheaper hardware like the C.H.I.P. at some point. I think they have a good chance at running at full speed.
The biggest problem with the Raspberry Pi for this kind of stuff is that the software, which mostly consists of lightly optimized ports of old emulators written for Windows, holds the hardware back. If you compare instructions-per-clock of CPUs and fill rates of console GPUs from that era with the relevant hardware on even cheap modern mobile SoCs, the latter blow the former away. It's just that the games of that era pushed the hardware to its limits, while the emulators on the Pi don't even come close.
There's no fundamental reason why the N64 can't be emulated at full speed on that hardware. The VideoCore IV blows the RDP away in fill rate, after all. It's just that the available video plugins are old and aren't optimized for mobile GPUs. Additionally, Broadcom's drivers have a tendency to stall in inconvenient places and the emulator needs to work around that as well. But once those are fixed, most of the popular N64 games run beautifully.