It must surely be generic equipment management, you're not running experiments in physics that are > 30 years old, surely?
> 20,000 lines of code really doesn't seem like that much.
It's not "20k lines of code" but "n lines of code that grew organically over decades to 20k lines" - with the help of probably way more than 100 people who all are not trained as developers. I think it's a safe bet to say the current state only has a faint memory of being a consistent code base.
Once I spent close to a month hunting down a subtle but nasty bug in a number crunching module which was perhaps around 2k LoC. Writing code is very easy. Verifying and validating number crunching code is very hard and very time consuming.
A Ph.D. student or a young researcher might be more interested if a rewrite lets them run experiments using Amazon / CUDA / whatever for a few thousand dollars instead of spending scarce grant money on dedicated hardware.
http://www.nersc.gov/systems/cori/
I completely agree on the domain knowledge statement. It is 100% required more then the CS degree.
as someone who had contact with that codebase, do you have any insights as to why that is?
Was it the sheer size of the thing? Was it some nuance that Fortran had as an advantage over other languages? was the math just difficult to follow?
Genuine curiousity.