Compiling pypy is memory intensive, I don't believe it can be done in 2GB which is where most of the really affordable ARM devices seem to cap out.
But why do you have to compile PyPy on the same machine you’re testing it on?
Well cross-compiling (aka "cross translation) is made complicated because PyPy doesn't use a normal toolchain, but that's not the problem here. The PyPy benchmark suite itself takes that much memory to run.
I would expect benchmarks to match typical usage, if PyPy can't run in a couple of GB then choose a better language.