Re:How to support open-source software and stay sane (2019)
by dekhn on Wednesday August 03, 2022 @10:29PM
(#32338657)
Although I totally understand how tough it is for a modern academic to maintain software, I gave some thought when I reached the Berkeley computational biologist who complained: Nelle Varoquaux says that, in her field of machine learning in biology, software tools quickly become obsolete because the size of the data sets is changing so rapidly
Let us consider another software project at Berkeley: BSD UNIX. BSD was a research project, a vehicle for groups to do operating system research, an environment for the development of modern relational databases, and the tools that it helped introduce have scaled through multiple orders of magnitude of data set size and through many different hardware architectures. It was done by a combination of professors who obtained funding and crazy grad students who lived and breathed UNIX.
Eventually, the primary funder felt the project had reached its impact limit and reduced funding https://www.tech-insider.org/unix/research/1992/0622.html but the reality is that all this open source stuff happened before and will happen again. The government funding agencies have a messed up incentive system that funds primarily the "sexy" stuff and only around the time when it's producing the "sexy" publications. For example, one of the most important things that kept BSD going was Ingres (oh look, berkeley CS still hosts a copy from 1989: http://s2k-ftp.cs.berkeley.edu/ingres/vax-bsd/)
The UNIX tools I've used for over 30 years have scaled to every IO system I've worked with until I hit multiple-terabytes and moved to parallel computing. Even today, you can pump terabytes through pipes and if you have enough space, you can sort enormous files.
Actually, some bio software also qualifies. AMBER has been around since the 70s and is still a competitive molecular dynamics code that runs on the largest supercomputers and fastest GPUs.
I've concluded that the best source for longterm funding is to convince some billionaire that bio software is important, and then get enough funding to fund an ongoing staff that maintains the software and supports the community, and spinning off a commercial company that handles the enterprise integrations.
Let us consider another software project at Berkeley: BSD UNIX. BSD was a research project, a vehicle for groups to do operating system research, an environment for the development of modern relational databases, and the tools that it helped introduce have scaled through multiple orders of magnitude of data set size and through many different hardware architectures. It was done by a combination of professors who obtained funding and crazy grad students who lived and breathed UNIX.
Eventually, the primary funder felt the project had reached its impact limit and reduced funding https://www.tech-insider.org/unix/research/1992/0622.html but the reality is that all this open source stuff happened before and will happen again. The government funding agencies have a messed up incentive system that funds primarily the "sexy" stuff and only around the time when it's producing the "sexy" publications. For example, one of the most important things that kept BSD going was Ingres (oh look, berkeley CS still hosts a copy from 1989: http://s2k-ftp.cs.berkeley.edu/ingres/vax-bsd/)
The UNIX tools I've used for over 30 years have scaled to every IO system I've worked with until I hit multiple-terabytes and moved to parallel computing. Even today, you can pump terabytes through pipes and if you have enough space, you can sort enormous files.
Actually, some bio software also qualifies. AMBER has been around since the 70s and is still a competitive molecular dynamics code that runs on the largest supercomputers and fastest GPUs.
I've concluded that the best source for longterm funding is to convince some billionaire that bio software is important, and then get enough funding to fund an ongoing staff that maintains the software and supports the community, and spinning off a commercial company that handles the enterprise integrations.