This is a really key point that is lost on devs outside of science looking in. In our case, good devs are out of budget by a factor of 2x at least (at an EU public university in a lab doing lots of computational work).

The best we get are engineers which are expected to keep the cluster running, order computers, organize seminars.. and eventually resolve any software or dev problems. This doesn’t leave much time for caring about reproducibility outside the very core algorithms. The overall workflow can fade away since the next post doc is going to redo it anyway.

The big issue with hiring software developers is that the 'payscale' is set according to the academic criteria, and an external developer coming from the industry - no matter how experienced or skilled - can usually be offered only a junior position with pay appropriate for that because they do not meet the criteria required for non-junior positions (no PhD, often not even a masters', no relevant publications, etc). From that perspective the only difference between a grad student who just started and a seasoned software developer is that the grad student can be employed as a part-time research assistant while a 'pure' developer could be full-time; but the hourly rate and conditions would be pretty much the same, targeted at less experienced employees. We can hire skilled mid-level individual contributors with reasonable pay for post-doc positions, but post-doc positions are limited to candidates who have a PhD. And it's not a that big limitation, since it's expected that everybody who's working "in the field" will get a PhD during their first few years of practical work experience as a grad student, the concept of "experienced/skilled but no degree" is not considered by the system as such people are rare in academia, and they stay rare due to the existing system.

So the disparity in evaluation criteria means that it's tricky to transfer between the different "career paths" - if you come from an environment where degrees mostly don't matter to an environment where a PhD is almost table stakes (to be a "hiring scientist", PhD is mandatory but nowhere near sufficient), then "getting your worth" is possible only if you are willing to put in quite some time and effort to fit the criteria used to evaluate scientists, even if you're there just to do software development.

My point is that it places outsiders (no matter if they're going to do software development or something else) on a lower payscale until they catch up on all the academia-specific factors of evaluation.

It's not a caste system between different types of activities, but rather a barrier of entry - in some sense, you have to start from 'level 1' no matter how much experience you have in other fields, so inexperienced people can join easily, but for senior/experienced people doing it is possible but costly.

There are many problems with using a dedicated programmer, or any other technical specialist in a small R&D team. The first is keeping them occupied. There was programming to be done, but not full time. And it had to be done in an extremely agile fashion, with requirements changing constantly, often at the location where the problem is occurring, not where their workstation happens to be set up. Many developers hate this kind of work.

Second is just managing software development. Entire books have been written about the topic, and it's not a solved problem how to keep software development from eating you alive and taking ownership of your organization. Nobody knows how to estimate the time and effort. You never know if you're going to be able to recover your source code and make sense of it, if your programmer up and quits.

With apologies to Clemenceau, programming is too important to be left to the programmers. ;-)

This is a general problem we all have, whenever we should employ a professional to do necessary work. The right professional will take a tenth of the time and the job will be done some multiple better. But how do you pick the right person?

I have two experiences with post-grad work that I think are relevant:

1. A friend needed some work done in a statistics package that used a language that felt like it was from the 80's. I was able to complete the work in a few hours, but I don't think a student could have done it (complicated need combined with a crappy language and IDE).

2. Another postdoc engineering friend needed to do some heavy duty data analysis, and she was recommended to learn C++. I suspect she wasted years learning C++, time which should have been spent on investigating different forms of analysis. She wanted to listen to her engineering fellows, not some practicing software engineer, so wasted her life not achieving much...