Some colleagues and I edited a special issue of a journal that had only papers-about-or-with-software and it was pretty well-received. That way I got to do service to my software-heavy colleagues too, with decent credibility due to my own well-known software. Tenure points all round.
I'll refrain from writing a wall of text on this, but my main point would be that the current objective function largely has a blindspot (open source software as intrinsic scholarship), and that overloaded people tend to be forced to optimize their critical path to "success". And software is decidedly not in the critical path, as you have mentioned.
Thanks, this (and your other answers) is very useful. I'm about to finish my PhD, and have a recently open-sourced code I've written that's also a bit tangential to my PhD work (it's for molecular dynamics, "raaSAFT" on Bitbucket/PyPi). It may be easier for this type of code, though, in the sense that it's an application rather than a library, but I am aware that I may have to down-prioritize working on it in the short-term. (I'm not in CS either.)
But, on a serious note, when I asked this point-blank at an orientation for new professors, the response from a senior EE professor was "never work on anything that won't lead to a publication". Unfortunately, as you're well aware, that isn't how open source projects work, but that is the perspective of many senior academics outside of my area (and likely software altogether) that will have loud opinions around tenure time.
I'm well aware that this is what I've signed up for, but I'm also keenly aware that many postdocs that are both academically and software-engineering capable can easily convince themselves that this isn't the cold truth.