I am the lead author of Elemental and am surprised to find this here. I'll try to answer any questions.

I understand (correct me if I'm wrong) that Elemental started somehow as a by-product of your PhD work. How did you find keeping working on it after the thesis, and growing a small team? Anything in particular that you think was important for this to succeed (I mean the keeping-on part, not the technical part)?

The brutally honest answer would be that I prioritize its development above everything else. I have no idea if this is sustainable in the long term, but I don't plan on slowing the development down as long as I have an income.

When I was finishing my PhD I turned down several industry opportunities for fear of having to stop development. I am currently a (tenure track) assistant professor but increasingly worry about academia's lack of respect for library development.

At this point, I don't think it'll help you, but about a decade ago, COIN-OR was put together to partially provide an organization that helps give academics a place to peer review and receive credit for the codes they generate (http://www.coin-or.org/). The other reason was to provide a place to host open source optimization codes before things like Github became popular. Now, did it every really live up to that promise? Not as much as I'd like. Honestly, after IBM purchased ILOG, it looked like their support dropped off pretty significantly and IBM was always the largest backer. That and a number of key people retired. In any case, there was just an election, but I know they're still looking for people to help drive the organization forward. As an assistant prof, it's a risk, but perhaps one worth looking at at least in the sense of what some kind of organization with moderate backing looked like that was trying to give academics better credit for their codes.

This is a great point. I'm an outsider in the optimization and OR communities (having started out in HPC linear algebra) but have a lot of respect for what I've seen from COIN-OR.

Library development is respected in CS, as long as you have a real community using the code so you can demonstrate impact. AND as long as you also graduate some grad students, write some papers, win some grants, do some service, and teach some classes decently well. You signed up for those things with the job and you can't substitute library development for any of them. It's up to you if you want to do it as well, and it might even be the best way for you as an individual to make impact, but you can't skip any of the traditional things.

Well, you could try, but it's pretty risky. My advice is to get the other things done enough that you can't be criticized for them, and also kick ass in software if that's your passion. No one said it would be easy.

/end unsolicited advice from greying software-heavy professor.

Too bad I'm not in a CS department.

But, on a serious note, when I asked this point-blank at an orientation for new professors, the response from a senior EE professor was "never work on anything that won't lead to a publication". Unfortunately, as you're well aware, that isn't how open source projects work, but that is the perspective of many senior academics outside of my area (and likely software altogether) that will have loud opinions around tenure time.

I'm well aware that this is what I've signed up for, but I'm also keenly aware that many postdocs that are both academically and software-engineering capable can easily convince themselves that this isn't the cold truth.

Can you write solid papers about the software? If it's original and impactful, you ought to be able to write about it. Journal editors can be happy to receive papers about popular and widely-used software, since the articles are likely to be cited a lot. Your happy users write nice reviews and everyone wins.

Some colleagues and I edited a special issue of a journal that had only papers-about-or-with-software and it was pretty well-received. That way I got to do service to my software-heavy colleagues too, with decent credibility due to my own well-known software. Tenure points all round.

There are definitely top-notch journals largely serving this purpose. And I know several academics tenured via such a strategy.

I'll refrain from writing a wall of text on this, but my main point would be that the current objective function largely has a blindspot (open source software as intrinsic scholarship), and that overloaded people tend to be forced to optimize their critical path to "success". And software is decidedly not in the critical path, as you have mentioned.

> many postdocs that are both academically and software-engineering capable can easily convince themselves that this isn't the cold truth.

Thanks, this (and your other answers) is very useful. I'm about to finish my PhD, and have a recently open-sourced code I've written that's also a bit tangential to my PhD work (it's for molecular dynamics, "raaSAFT" on Bitbucket/PyPi). It may be easier for this type of code, though, in the sense that it's an application rather than a library, but I am aware that I may have to down-prioritize working on it in the short-term. (I'm not in CS either.)

Also, I should clarify that the library was originally orthogonal to my PhD work and done at home on my laptop at night. The very early version of the library was about a summer's worth of full time (unfunded) work. I didn't start absorbing it into my research until the end of my Ph.D.

Tldr from the website: Elemental is an open-source library for distributed-memory dense and sparse-direct linear algebra and optimization which builds on top of BLAS, LAPACK, and MPI using modern C++ and additionally exposes interfaces to C and Python (with a Julia interface beginning development).

It'll be interesting to see the how the Julia interface behaves and how big the performance penalty is going to be.

The idea has been to use the naive (in the sense of being distribution agnostic) batched interface to filling/querying Elemental's distributed data structures to interface with Julia's `DArray`. This essentially boils down to one all-to-all communication of the row, column, and values of each update and should have a runtime that looks like O(n^2/p) + O(p) for n x n matrices over p processes. Ideally one performs such redistributions as few times as possible if wanting to use routines that only require quadratic work or less (such as matrix-vector multiplication); for dense factorizations the overhead should be reasonably ignorable.

PETSc has an Elemental interface for doing dense linear algebra. Can't do much better for an endorsement than that.

I'm definitely happy that the PETSc developers have put up with Elemental being such a moving target (it has expanded from dense factorizations to sparse-direct, Interior Point Methods, and now some number theory), as there have been substantial growing pains. Barry Smith has also been nice enough to poke me any time Elemental isn't valgrind clean.

When you say that you can do arbitrary precision linear algebra, do you control the scalar type or is the code templated on it? Basically, can we run the factorizations over arbitrary scalar types, which allows things like automatic differentiation over the factorizations?

How does the performance of the linear algebra compare to other templated linear algebra codes like Eigen?

As of a few days ago, Elemental supports a BigFloat class which wraps MPFR along with MPI wrappers and distributed matrix classes which template over it.

It would be a modest project to support other datatypes, as the main obstacles are MPI wrappers and perhaps some explicit instantiations. I would be happy to respond in excruciating detail on [email protected].

As for performance: Elemental is focused on distributed-memory functionality but still achieves performance comparable to optimized libraries when run on a single process (indeed, Elemental often maps down to OpenBLAS, MKL, etc. when run with only one process).

Does this overlap functionally with TensorFlow ([1])? How are the two different?

[1] https://www.tensorflow.org/

I would say that the goals align much more than the current approaches, which are somewhat orthogonal. My focus in developing Elemental has been on expanding from the scope of libraries like ScaLAPACK (which focused on distributed dense linear algebra) into more general algorithms (e.g., Interior Point Methods and lattice reduction) and modern programming practices (using templates to support arbitrary-precision versions of the above).

Arguably the biggest weakness of Elemental is its lack of focus on dynamic scheduling at an intranodal level, though this can to a large degree be pushed into a local BLAS layer. One of the big items on my wish list would be proper integration of intranodal dynamic scheduling with the current high-performance internodal static scheduling.

The current TensorFlow releases seem to be primarily focused on intranodal dynamic scheduling; there has been a substantial amount of work in this area from the PLASMA [1] and (less recently) SuperMatrix [2] projects.

[1] http://icl.cs.utk.edu/plasma/ [2] http://www.cs.utexas.edu/users/flame/pubs/SuperMatrixTR.pdf

How about dask's distrubted array? Do you know how the architecture and goals compare to Elemental?

Distributed [1] is very new and seems to have similar core architectural goals as TensorFlow. But perhaps I'm being too politically correct: both make use of very course-grain parallelism relative to a typical distributed-memory linear algebra library (e.g., the current communication mechanisms of both are likely to be too course-grain to efficiently support distributed dense matrix inversion or eigensolvers; not that this is likely to be a design goal of either).

[1] http://matthewrocklin.com/blog/work/2015/06/23/Distributed/

I think distributed's scheduler has seen much iteration since then and is now based on tornado. I wonder if that changes your assessment.

Also what do you think of Julia's native distributed capabilities and this library here: https://github.com/shashi/ComputeFramework.jl