Indeed, we didn't chose Elemental over ScaLAPACK; we used Elemental to replace LAPACK because it was impossible to refactor our code to use ScaLAPACK without absolutely ruining it forever. The ScaLAPACK interface is horrible and should only be used inside of a higher-level library interface that hides its ugly, ugly control structures.

From my perspective, Elemental is primarily about human performance. That it scales to thousands of nodes of Blue Gene and achieves performance on par with (and in some cases significantly better than) ScaLAPACK is really icing on the cake. Granted, I really like scalability and performance, but I will not ruin my code in the name of these things.