Interesting, would love to hear some of your experiences! It sounds like the issues we face are slightly different. From my field, version control is pretty much a non issue[1], most of the widespread codes use that (typically svn, but that's fine). Build systems are often an issue, as is incomplete (or outdated) documentation, and a lack of proper changelogs. A lack of proper tests too. Actually, maybe the issues we face aren't so different after all...

> The solution already exists: skilled scientist hires grad student who's done some programming + some electronics classes. Happens everywhere already, works well.

I've never actually seen this work well. Scripting an existing solution, sure. But writing a new one from scratch... Most scientists I have met personally who code, write awful awful code. Sometimes they just lack the time to do something better (I'm as guilty of this as anyone), other times they're just not very good at it. And it's almost never maintainable: when they leave the project, you might as well re-write from scratch (a more cynical person would say that's by design, but I actually don't believe that). There are many, many exceptions to this, obviously, but as a rule...

I am optimistic though. It seems like things are changing for the better, slowly.

Edit: [1] That said, I've been trying to persuade a colleague to use version control for ages. I'm at a loss. Live and let live, I guess.

(What's you field? Mine's computational physics, but I collaborate a lot with experimentalists.) When I say version control, I meant for in-house projects, that tend to be just on someone's laptop and also a horrible mess, as you say. Most public codes are indeed on some VCS, git is very popular in my field. But reproducibility is a whole other matter. There's been some interesting work on providence systems that can attach all input needed to run a code for reproduction to e.g. figures in a paper, but until we've got a good universal one that journals start caring about, they're not going to see wide adoption.

As for non-computer-savvy profs using grad students for programming: I see a lot of either LabView (for controlling experiments) or Python/Matlab in those cases, and it's often "write-only" code, but it mostly gets the job done. I'm also optimistic, but there are too few people working on and caring about the software carpentry we need to back modern svience in a good way.