I think a big difference is the flags. At least for g++, if you don't specify -march=native -mtune=native you're going to take a performance hit. How much of a performance hit depends on the features.

The sorttest.zip Makefile has only the following flags specified: -O3 --std=c++11

Where bjourne has: -O3 --std=c++11 -fomit-frame-pointer -march=native -mtune=native

I might rerun your tests with bjourne's additions!