It's generally not plausible to "approach someone with familiarity with software development expertise" for organizational and budget reasons. Employing dedicated software developers is simply not a thing that happens; research labs overwhelmingly have the coding done by researchers and involved students without having any dedicated positions for software development.

In any case you'd need to teach them the problem domain, and it's considered cheaper (and simpler from organizational perspective) to get some phd students or postdocs from your domain to spend half a year getting up to speed on coding (and they likely had a few courses in programming and statistics anyway) than to hire an experienced software developer and have them learn the basics of your domain (which may well take a third or half of the appropriate undergraduate bachelor's program).

> Employing dedicated software developers is simply not a thing that happens

This is a really key point that is lost on devs outside of science looking in. In our case, good devs are out of budget by a factor of 2x at least (at an EU public university in a lab doing lots of computational work).

The best we get are engineers which are expected to keep the cluster running, order computers, organize seminars.. and eventually resolve any software or dev problems. This doesn’t leave much time for caring about reproducibility outside the very core algorithms. The overall workflow can fade away since the next post doc is going to redo it anyway.

Are the hiring scientists also paid well-below market wages by that degree?

In many fields industry pays noticeably better than academia, but the difference is not that meaningful for the actual scientists; a principal investigator gets a reasonable amount of money but also significant degree of freedom and influence which helps job satisfaction even if the pay itself is lower.

The big issue with hiring software developers is that the 'payscale' is set according to the academic criteria, and an external developer coming from the industry - no matter how experienced or skilled - can usually be offered only a junior position with pay appropriate for that because they do not meet the criteria required for non-junior positions (no PhD, often not even a masters', no relevant publications, etc). From that perspective the only difference between a grad student who just started and a seasoned software developer is that the grad student can be employed as a part-time research assistant while a 'pure' developer could be full-time; but the hourly rate and conditions would be pretty much the same, targeted at less experienced employees. We can hire skilled mid-level individual contributors with reasonable pay for post-doc positions, but post-doc positions are limited to candidates who have a PhD. And it's not a that big limitation, since it's expected that everybody who's working "in the field" will get a PhD during their first few years of practical work experience as a grad student, the concept of "experienced/skilled but no degree" is not considered by the system as such people are rare in academia, and they stay rare due to the existing system.

So the disparity in evaluation criteria means that it's tricky to transfer between the different "career paths" - if you come from an environment where degrees mostly don't matter to an environment where a PhD is almost table stakes (to be a "hiring scientist", PhD is mandatory but nowhere near sufficient), then "getting your worth" is possible only if you are willing to put in quite some time and effort to fit the criteria used to evaluate scientists, even if you're there just to do software development.

Academia definitely can place software creators at a lower payscale to keep other higher. I wonder why such a caste system exists?

It does not place software creators at a lower payscale - all the software creators I know in academia are at the payscale level where they should be given their experience, however, all of them have a PhD or are in the process of getting one very soon.

My point is that it places outsiders (no matter if they're going to do software development or something else) on a lower payscale until they catch up on all the academia-specific factors of evaluation.

It's not a caste system between different types of activities, but rather a barrier of entry - in some sense, you have to start from 'level 1' no matter how much experience you have in other fields, so inexperienced people can join easily, but for senior/experienced people doing it is possible but costly.

It’s not a caste system, rather a capitalist imitation where the capital is impact factor and grants and first authorship in nature and science. In this system software creators are just means to an end, and good software engineers are an irrational cost given the PhD student that can churn out working code for same impact factor for less money.

As a grad student in physics, I not only wrote code, but also designed my own (computer controlled) electronics, mechanics, optics, vacuum systems, etc. I was my own machinist and millwright. Today I work in a small R&D team within a larger business, and still do a lot of those things myself when needed.

There are many problems with using a dedicated programmer, or any other technical specialist in a small R&D team. The first is keeping them occupied. There was programming to be done, but not full time. And it had to be done in an extremely agile fashion, with requirements changing constantly, often at the location where the problem is occurring, not where their workstation happens to be set up. Many developers hate this kind of work.

Second is just managing software development. Entire books have been written about the topic, and it's not a solved problem how to keep software development from eating you alive and taking ownership of your organization. Nobody knows how to estimate the time and effort. You never know if you're going to be able to recover your source code and make sense of it, if your programmer up and quits.

With apologies to Clemenceau, programming is too important to be left to the programmers. ;-)

There's no problem with not leaving programming to programmers, its about how to encourage anyone picking up programming to have healthier habits so others can participate in the creation in the future.

Indeed, and one thing that's lacking is any kind of coaching or training. Those of us doing it can't necessarily coach the next generation, because stuff has gotten ahead of us.

> research labs overwhelmingly have the coding done by researchers and involved students

This is a general problem we all have, whenever we should employ a professional to do necessary work. The right professional will take a tenth of the time and the job will be done some multiple better. But how do you pick the right person?

I have two experiences with post-grad work that I think are relevant:

1. A friend needed some work done in a statistics package that used a language that felt like it was from the 80's. I was able to complete the work in a few hours, but I don't think a student could have done it (complicated need combined with a crappy language and IDE).

2. Another postdoc engineering friend needed to do some heavy duty data analysis, and she was recommended to learn C++. I suspect she wasted years learning C++, time which should have been spent on investigating different forms of analysis. She wanted to listen to her engineering fellows, not some practicing software engineer, so wasted her life not achieving much...

> I would have thought the scientists among us would approach someone with familiarity with software development expertise.

Is there a pool of skilled software architects willing to provide consultations at well-below market wages? Or a Q&A forum full of people interested in giving this kind of advice? (StackOverflow isn't useful for this; the allowed question scope is too narrow.) I guess one incentive to publish one's code is to get it criticized on places like Hacker News. The best way to get the right answer on the internet is to post the wrong answer, after all.

I'll state the obvious and answer with No. There are not enough skilled software architects to go around and many who consider themselves skilled are not actually producing good code themselves, probably including many confident posters here in this forum.

The idiosyncrasies and tastes of many 'senior' software engineers would likely make the code unreadable and unmaintainable for the average scientist and possibly discourage them from programming altogether.

Software architecture is an unsolved problem as evident in the frequent fundamental discussions about even trivial things, highlighted by a cambrian explosion of frameworks who try to help herding cats, and made obvious in senior programmers struggling to get a handle on moderately complex code.

I propose scientists keep their code base as simple as possible, review the code along with the ideas with their peers, maybe use Jupyter notebooks to show the iterations and keep intermediate steps, and, as others state, show the code as appropriate and try to keep it running. There is no silver bullet and very few programmers could walk into your lab or office and really clean things up the way you'd hope.

I think the suggestion to keep the codebase as simple as possible for scientists applies as well to the software creators.

Life is different when you might have the a relationship with a single code base for 2-5 years, or even more. Complexities will happen on their own, no need to add them in.

Are the hiring scientists also paid well-below market wages?

> Are the hiring scientists also paid well-below market wages?

Yes. Well, in engineering anyway. That's why most engineers use academia as a stepping stone to something else. Working in science is, I think, sort of like working at a startup that's perpetually short on cash with no possibility of an exit.

With a little digging you can find published tables of job codes and salaries for many universities, e.g., https://www.udel.edu/content/dam/udelImages/human-resources/...

Faculty are listed separately for some reason: https://www.udel.edu/faculty-staff/human-resources/compensat...

Positions requiring a PhD start being listed at 29E (midrange of $55k) or 30E (midrange of $63,800). You could easily get that with a bachelor's degree in engineering 10 years ago. I suspect you will find the "Information Technology" Job Family salaries particularly amusing.