You think web developers are just writing programs for the hell of it? They have operational constraints around their work as well, it's just that there's a rad open online culture of continuous process improvement that's leveling up the tooling and practices that makes what was once hard look easy.
Things scientists using computers can learn from people working in the software industry:
- use a version control oriented workflow
- write extensive tests
- data management, automated data processing/cleaning pipelines, backups
- build generic frameworks
- use continuous integration
- logging and error reporting for long running tasks
- use modern tooling for automating infrastructure and big jobs, rather than manually submitting Slurm jobs via SSH
#notallscientists but the average level of competence is not good and I think the idea that writing software for scientific research is somehow special is backwards and counterproductive.
I don't see that many job posts doing this anymore. And even then it's just not valued that much.
Not all software development should involve writing your own framework, and it's a bit of a trope that junior SWEs try to write DSLs and frameworks where they should just write some throwaway code. That said, if there's a common modelling or simulation method in your field and there's no great open source generic framework for it, then you should take a hard look at what you are all spending your time on (endlessly reimplementing the wheel).
Is that going to be robust to whatever new cluster comes along five years down the line, or will it all have to be re-written?
You give good reasons for why a certain situation exists in scientific communities, but I see no reason why it has to be that way.
The various conventions around "accepted" programming paradigms like OO, inheritance, scoping, etc are natural if you use them every day for years, but if you're more interested in optimizing some finances or solving a physics problem, something "primitive" or "messy" like a spreadsheet formula or fortran might actually be more understandable.
I appreciate your point. In most contexts, such as your comparison between Japanese and English, I agree with it. However, paradigms like OO, inheritance, scoping, etc. are hard-won, intellectual accomplishments; they're not arbitrary. They're purposefully designed to solve specific problems. My experience with R showed it to be rife with problems that have been avoidable for decades. AFAICT, it boils down to the tragic view of "I'm a (statistician|engineer|mathematician) not a coder so I don't need to care". The unfortunate truth is that despite such a view, doing analysis with R makes the analyst a programmer by definition. And so the language, ecosystem, and, consequently, users suffer from half-baked, poorly designed workarounds to problems which have long been solved (and abundantly documented in the software literature). To reduce it to a matter of perspective feels to me like a tautology: it's easy once you get it.
Clearly, I'm triggered by this. I hope I've expressed myself respectfully. My point is, I don't feel it's arbitrary. The design and complexity of R has real consequences in terms of cost and reproducibility.
R does have inheritance by the way, not as you will see frequently on "general purpose" programming languages, thing R is not.
[1] http://catern.com/inheritance.html [2] https://books.google.pt/books?id=ka2VUBqHiWkC&pg=PA81&lpg=PA...
"Inheritance" is really the expression of Abstraction in code. Superclass is more general, and thus the abstraction of its subclasses.
Although Smalltalk did not have a specific keyword for "Abstract Class" it used the convention of calling "subclassResponsibility" on methods defined in abstract subperclasses which had to be implemented in subclasses.
Abstraction, generalization, these are conceptual tools. I'm not sure how "composition" models abstraction if at all. Only OOP does.
And one could think that abstraction is a tool valued by scientists. No?
Everything I was taught when OOP became mainstream in the late 80s, was that it was a performance trade off to afford code organization insight, while preserving encapsulation.
This statement about “performance reasons” is quite baffling.
I disagree. There are inherent qualities associated with systems that happen to be easier to understand. If you or I were designing such a system, and wanted to ensure this system imbued similar qualities — I believe it would be possible to do so.
> Japanese is "difficult to understand" if you only know English, but not if you grew up speaking Japanese.
Needing a priori knowledge to be able to understand something, is different.
Can you formalize these inherent qualities? Until you can, it's hard to prove that a quality is or is not essential.
There is an obvious one, though it's tricky to fully formalize and quantify: locality (also known as "coupling"). You ask yourself a question, "if I were to make a small change to a single piece of this program, how much of the program would I have to change/retest/keep in mind?". If the answer is, "typically, just the area around the change", it's a high locality (loosely coupled) program, a good design. If the answer is, "usually, most/all of it", then it's low locality (tight coupling), a bad design.
The reason is, of course, that human mental capacity is limited and in general, number of interactions grows superlinearly with the amount of interacting components. So this here is an objective measure of "easy/hard to understand": the more code you have to keep track of when investigating a random piece of the program, the more mental effort you have to exert - the more difficult to understand it is.
It has some comments about why good software engineering leads to good software, how good software engineering is hard right now because there's a lot of bad examples which cause people to make more bad code, and some suggestions on how things could be done.
Especially consider that much of the audience for R come from proprietary and extremely expensive languages like SAS or SPSS Syntax, neither of which are significantly less difficult to work with but significantly more limited.
Something more intuitive would be nice, but I'm not aware of another option that has the depth of libraries for practical every type of data analysis that can also begin getting a researcher meaningful results in few lines of code than they have fingers on their hands. (Assuming there was no catastrophic accident resulting the loss of fingers.)
Almost all the progression in that industry is relatively straight forward and would make sense to the lay person.
The issue isn't just specialized knowledge - knowing the principles underlying the process, knowing that there are principles - but also access to tooling. I'm sure plenty of lay cooks would appreciate appliances that let them be more consistent, even without knowing any related formal theory - but these are not available. As it is, an oven that's not shit is a major capital investment. It only makes sense to get one if you're more likely to call your work output a "formulated product" than "cookies".
However, software development is a peculiar occupation - one in which the best tools are free. It's like being able to buy a production line for less than a consumer-grade stove. Access to quality tools is not a limiting factor. Knowledge and awareness is.
Of course, non-programmers that code have more interesting things to do than to study software engineering. Which is why it's doubly important to make sure tools that are being popularized aren't shit. On the contrary, tooling is the perfect place to encode good practices and principles, so that even most lay (programming-wise) users succeed by default.
And they don't value writing programs. Or Software Engineers.
That makes productizing some research interesting. That also makes trying to get the same result as something published by your own lab a year ago an uphill battle: "it worked on John's old laptop, the huge Alienware. Never worked on any of our machine".
Its something like how baking cookies at home and running a cookie factory are very different. To a person doing each, the behaviors and priorities of the other seem strange and it’s easy to for one to think “we are both just making cookies, why don’t they do what i do which is obviously superior “
Scientific programmers are solving problems first, not writing programs. They are solving problems in a way that is useful only to them or peers who know a whole lot about the problem being solved. The problem-first tools look strange because they deemphasize the programming niceness in favor of problem niceness.
You find the same sort of confusion when programmers are facing business types and excel usage.
There are certainly times when a piece of code starts needing the programming touch, but the right tool for the job depends on the job.