Based on perf report, the latest stable version of exa is spending >99% of its time doing grid stuff. I pulled the repo from git, and using the latest master commit, it appears to have a huristic to not bother with the grid when you have lots of files. Now the times are a lot closer. (redirected stdout to /dev/null since otherwise the vast majority of the time would simply be outputting to the shell)

    [user@anarchy:~/exa_test]$ time ../exa/target/release/exa >/dev/null                                                                                                                                                                                                      
    ../exa/target/release/exa > /dev/null  0.05s user 0.07s system 99% cpu 0.122 total
    [user@anarchy:~/exa_test]$ time /bin/ls >/dev/null                                                                                                                                                                                                                        
    /bin/ls > /dev/null  0.03s user 0.01s system 99% cpu 0.048 total