"At the time, Randy was still stupid enough to be flattered by this attention, when he should have recognized them as bone-chilling words. Three years later, he left the Astronomy Department without a degree, and with nothing to show for his labors except six hundred dollars in his bank account and a staggeringly comprehensive knowledge of UNIX."
– Neal Stephenson, _Cryptonomicon_ (p. 78). HarperCollins.
To paraphrase: "Usually a comprehensive knowledge of UNIX and a girlfriend are mutually exclusive."
But it describes two options:
1. "A staggeringly comprehensive knowledge of UNIX", three years of domain-specific education, and a network of people who trust you to--no, who depend on you to be able to get things done with that knowledge
2. A piece of paper just like everyone else in the department.
And while the former might be more difficult to make use of, I think it could be much more valuable in the long run.
1. "A staggeringly comprehensive knowledge of UNIX", three years of domain-specific education, and a network of people who trust you to--no, who depend on you to be able to get things done with that knowledge
2. A piece of paper just like everyone else in the department, three years of domain-specific education, and a network of people who trust you to--no, who depend on you to be able to get things done with that knowledge (and probably a girlfriend)
This might be the case, but he's now like the janitor who unplugs the toilet. His work may be appreciated, and it might be necessary, but it's not respected or well remunerated. I guess he gets a little bit of respect for unplugging more theoretical pipes, but a little respect isn't a PhD.
>> except six hundred dollars in his bank account >> are ways to get this knowledge.
I managed to hack in a step which just dumped the memory of resulting frame to a file, and then we wrote a Python script to read that and produces a PNG.
We combine the PNGs into an animation and show someone else in the department because the supervisor wasn't in that day. "Cool! But, hrmm, those boundary conditions look wrong." Sadly, this was 2/3s into his Masters.
Like much now in Physics, the original code was simply incorrect. This happens all the time, and papers get retracted because of it - well, that's the best case if somebody notices.
At the end of the day, programming languages come with eco-systems, and must be chosen solely as a tool. The problem is not with Fortran, but that often Fortran == outdated development practices and in Physics horrible code hacked on by 10+ people without prior programming experience.
I don't know what can be done about it except hiring programmers to write code, which wouldn't be either easy or cheap
I can see many projects being improved by providing a Python pre-processor that writes out e.g. a binary config file, the hard-code Fortran/C simulation code reads that, and spits out the simulation results, and then having a Python post-processor that does the pretty stuff at the end.
Academically, it seems that pairing CS undergrads with Physics undergrads to do e.g. a molecular dynamics (MD) course would be cool. The Physics behind MD isn't too hard, and given the right parameters the programming part would be manageable. Then again, CS undergrads aren't necessarily great programmers either...
Clean code and tests are overrated. Version control and a big eco-system is underrated. It's like maths, physicists don't understand maths, they just use it like a carpenter uses a nail without understanding metallurgy, or a programmer uses a CPU without understanding solid-state physics. And that's okay.
But that doesn't seem the case. The choice seems to be between python and c++/fortran. Does anyone know why?
2. The choice is either "fastest possible" or "I just don't care how long it takes" - for either development or run time. There is usually no middle ground.
3. Tooling - the scientist is most likely using a text editor not an IDE (especially when they start working). Fortran and python are both low enough on boilerplate to not need IDE support.
4. Abstraction level. Scientists in general don't bother care about abstraction level at all. Procedural programing's abstractions are usually more than enough for them (and may actually be the correct level for some numerical work - think cache misses vs. hits).
5. Some areas of science do use Java (check out imagej for example).
There are several ways to bind Python and fortran together. E.G http://arogozhnikov.github.io/2015/11/29/using-fortran-from-...
Sounds perfectly normal :)
The standard advice here at work for one of our modelling packages is: if it crashes try rotating your input model by one degree and try again. Most of the time that will fix it. If it doesn't fix it try one degree in the other direction.
Is correctness somehow assured provided it doesn't crash? Or is the whole thing just a joke?
I'm sure lots of people do. We're a commercial company selling consultancy service around this software (technically around some inhouse tools built on top of this software), among many other things.
Is correctness somehow assured provided it doesn't crash?
On the whole well tested PDE solvers either crash, give answers that are off by several orders of magnitude, or give a correct answer. (If the answer they give is relevant to what you're trying to model is left as an exercise to the reader)
We're reasonably sure that, if the calculations converge, the solution provided is a correct numeric approximation (within the error bounds given) of the PDEs we're claiming to solve. We also believe that the PDEs we're using provide a reasonable balance between modeling what we claim we're modelling and our computation running in a reasonable amount of time.
In contrast, porting the 30+ year old molecular Dynamics package AMBER to CUDA took about 3 months and probably established me as a CUDA expert. In my opinion its well-maintained Fortran 90 code was far easier to understand and refactor.
While my primary languages today are C++ and CUDA, there is something clean about Fortran when it comes to understanding underlying algorithms. I have a similar opinion about well written C code.
The compression would presumably remove the redundancy of the language itself as a factor (including differences in idiomatic cyclomatic-complexity "depths" of various stdlibs), and also remove any redundancy in the way the test cases were specified. So it'd be down to a measure of how much circumlocution and over-engineering you did in the process of implementing the solution.
I'd worry slightly that code-golf solutions would be rewarded, though. Maybe pass everything through an obfuscator + linter before computing the metric, so that things like identifier lengths and spaces aren't considered.
I constantly run into code where someone takes an obscure corner case in the language, and builds an entire edifice around it.
https://sourceforge.net/projects/folderol/
Why write the port in the first place? Because back in the days of yore, I was doing the equivalent of adversarial search to try and design a better potential function for predicting protein tertiary structure. I ultimately arrived at the result that there were too many adversaries to make linear models and single hidden layer neural networks work.
And unfortunately, my postdoctoral advisor at the time didn't consider this publishable research.
OO allows you to make a much worse mess than procedural programming in my experience.
May I ask how? I don't see ads or any other obvious monetization channels.
GP is not talking about http://rosettacode.org/wiki/Rosetta_Code
The head of information security at the UW calls the PSTN bulletproof secure, but VOIP insecure, then babbles on about how he shares classified info with his buddies casually (happens to be a felony to share said info).
Needless to say, UW's Avaya phone system barebacks the internet, with no regard for using silly security things like TLS or SRTP.
I would guess that a C++ codebase written by PhD students, not by seasoned C++ experts, is more complicated and much slower to debug, than a corresponding Fortran codebase.
No, I'm not kidding.
I shudder to think what they'd have done with C++. :-)
Function has its side effects. It's use as only once case is really questionable, justified by lack of other means.
So especially in the hands of non-experts, Fortran should produce less bugs.
I don't know of any studies like this. But there are two aspects to consider. One is what you've pointed out, which is the long-term scientific gains and the productivity of the research group as an entity. But the other aspect is whether those students who rewrote the code into C++ were more or less successful than the students who weren't rewriting code.
Obviously, how you judge success in the latter will be tricky, since there may be differences in the interest of students who rewrote the code and their desired outcome after the PhD (e.g., stay in academia versus go into industry).
as someone who had contact with that codebase, do you have any insights as to why that is?
Was it the sheer size of the thing? Was it some nuance that Fortran had as an advantage over other languages? was the math just difficult to follow?
Genuine curiousity.
It must surely be generic equipment management, you're not running experiments in physics that are > 30 years old, surely?
> 20,000 lines of code really doesn't seem like that much.
It's not "20k lines of code" but "n lines of code that grew organically over decades to 20k lines" - with the help of probably way more than 100 people who all are not trained as developers. I think it's a safe bet to say the current state only has a faint memory of being a consistent code base.
Once I spent close to a month hunting down a subtle but nasty bug in a number crunching module which was perhaps around 2k LoC. Writing code is very easy. Verifying and validating number crunching code is very hard and very time consuming.
A Ph.D. student or a young researcher might be more interested if a rewrite lets them run experiments using Amazon / CUDA / whatever for a few thousand dollars instead of spending scarce grant money on dedicated hardware.
http://www.nersc.gov/systems/cori/
I completely agree on the domain knowledge statement. It is 100% required more then the CS degree.
IMO, Python stands a better chance of breaking the Fortran's lock on physics related computing. Give it a few more years and enough numpy-based libraries might make Python a real competitor. My 2c
Can you clarify why you think Python might be able to do it? For scientific computation with high performance requirements, Python is not competitive with Fortan or C++. For work that continues to happen in Fortran due to "academic inertia", my impression (and vague experience) is that researchers find the convenience of what they're accustomed to (Fortran) to be greater than the convenience of things like rapid prototyping offered by the various Python scientific computing and data analysis libraries. There is a mental overhead is switching, and I think most academics are sort of okay writing code in whatever is familiar and battle-tested if it means they can focus more on the research at hand.
In other words, I'm not saying you're wrong, but I'm not following your reasoning.
Speed is not nearly as important now as it was 10-15 years ago. A typical scientist's worstation has 20 CPU cores. If I want to use 100 CPUs for a few days, it is trivial and 1000 is easy to get. Thus the fact that Python is slow(er) does not bother me unless I am setting up something major.
What matters though, is the availability of libraries that let me reliably run my experiments. If there is a bug, it must be in my code, not the library -- a "discovery" caused by a software bug is humiliating. This is where Fortran shines and Python is not quite there yet -- the decades of beating on those libraries made them very well understood and trusted.
I am not talking about interactive programs -- a slow browser is annoying. I am talking about scientific computing. This is just my experience, can you provide some counterexamples?
Speed is not a huge issue if you're happy to leave your simulation running overnight anyway, or if you have the option to just throw more and more cores at the problem (or in our case, both). The goal is to get your papers published. Code developing/interpreting/debugging time is generally far more serious an obstacle to that goal than simulation speed. I was the only developer there. Everyone else in the group coded on an almost daily basis, but none of them considered themselves programmers, and very few of them actually learn about good programming practices. They're biophysics researchers.
Having not worked in Python much before I was also pretty pleased to find that, having determined that I needed to use Dijkstra's algorithm for path-finding and then working out the smallest Standard Deviation between certain sets of data points, Python came with libraries to do both of those things off the cuff. It's just so easy, I can see why it has a favoured place in this field.
What about those cases that it's the matter between throwing at 1000 core cluster and wether we can have results before next conference in a couple months. That is what really defines the scientific programming -- it's about feasible and infeasible. For other jobs, isn't it just matter of taste?
when I worked for BHR group (a Hydro dynamics research organisation) some times we had emergency projects that had < 24 hour turn around. I recall one where a clinet had had a serious issue at a plant and we ran the simulation and produced a report in a single day
A lot of Numpy is written in FORTRAN.
Graduate students need to think in terms of 4-6 years while forward looking professors might want to think in terms of 5-15 years.
However, I guess one drawback is that a lot of the things we currently implement are all written from scratch (for very standard things such as numerical differentiation and parameter optimisation), which has the advantage of having "control" and more understanfing over the code, but less time saving/potentially not as efficient as using pre-existing libraries.
In the process, they would have understood nearly every approach taken by the group towards producing the results that it does, and I bet that has helped them when they've ended up modifying that same code later for their research. And with fewer supervisor meetings to work out exactly what X, Y and Z part of the code does because they will have worked on it themselves.
I'd say I've easily spent 6 months just spending time getting my head around all the code we work with anyway, so at the very least, I hope that made their following years of research more productive.
A bad advisor wouldn't care what his students wanted, but a good advisor might still have students work on this kind of project as long as they were aware it wasn't going to help them get a tenure track position in the future. If they wanted to go work at a national lab doing HPC work and programing it might have been plenty ok for their career (This is what I am transitioning to now) or if they want to go work for a hedge fund or apple it might also be an okay option.
I wonder if there are any transcoders for Fortran to C++? I wonder if there is even a market for something like that? I've written alpha versions of transcoders for C to Java and Java to Objective-C and I think I could do the same for Fortran to another language but why?
Writing a transcoder first is the right way to do it and it should take you a couple of months to do it, especially if you already have some experience with this kind of stuff. Definitely less than a year for a single person. I would not try to translate the code by hand. It would be a never ending project full of bugs.
Edit: Apparently there are plenty of Fortran to C++ transcoders. Here is just one I found during my google search: http://cci.lbl.gov/fable/
No idea how good they work.
I did actually rewrite both IDL and Fortran, but it was always smallish, single-purpose programs or functions.
Note that the SPICE Toolkit[1] is still written in Fortran today, translated to C using f2c, and the C version used as the base for other language support. The Fortran part is unlikely to ever go away, since support for processing past missions is crucially important and all that processing code was written in Fortran. Also, talk about your stable, backward compatible APIs...
That being said, as someone who has integrated CSPICE into several C++ and python projects, the modernization would be a very welcome change. The current arch depends far too much on global state, and none of it is threadsafe.
Good to know Boris is still there! Was Chuck still heading up the team when you left?
> a plan for a complete rewrite in C++
Oh, no! That's a real shame. I'd be all for a rewrite, especially considering their insane test coverage, but that decision worries me.
> That being said, as someone who has integrated CSPICE into several C++ and python projects, the modernization would be a very welcome change.
As someone who worked mostly with CSPICE in C but also with some other non-Fortran toolkits, I agree.
I actually rang the recruiter to query why would not be simpler to train your existing staff to use Fortran - the add stayed up for years and years I always wonder if it ever got ported.
There's a salient lesson here. Clearly, this was a stupid decision for two reasons:
(1) there are many thousands of scientific routines that have had 40-60 years of fine honing and careful debugging and they just work! (For instance, we send voyagers—Voyager, Cassini, etc.—to the end of the solar system and they invariably get there; the Fortran routines that get them there do exactly what they're supposed to do (unlike much of today's poorly written C code)!
(2) Rewriting that already-reliable fully-debugged Fortran code into any other language will almost certainly make it far less reliable, thus it's a no-brainer to stick with the original .F source.
Yes, Fortran is simpler than C [consider it BASIC on steriods] and years ago it had long since evolved well past John Backus's 1954 incarnation of the language into solid workhorse that both physicists and engineers use regularly.
Just because something is old and out of fashion doesn't mean that it's broken or doesn't work well. (Longevity ensures that there's been sufficient time for many hands to make it reliable.)
(Oh, BTW, I'm reminded that decades ago when I was just beginning to learn Fortran using punch cards on an IBM360 mainframe with the WATFOR FORTRAN-IV compiler, that I made four errors in only six lines of code. After dutifully printing ERROR against each offending line, the compiler finished with the message: "YOU NEED TO SEEK ANOTHER CAREER " or words to that effect [yes, it was in uppercase]. Eventually, I got considerably better.)
From this point of view, rewriting code is an extremely high-risk proposition. As we know, the likelihood of discovering bugs during this process is quite high.
Curious if anyone in your group considered using it?
So cut-n-paste code. The students are running code that they don't really know what it does, it might not even be correct...it's like StackOverflow in academia.
This is so true. I'm a PhD student in physics using Fortran for pretty much that reason. At the start of my PhD, in response to my supervisor telling me I should learn Fortran to modify our current codebase, I asked if I could rewrite what I'd be working on into C++ first, since I was already familiar with it and wanted to bring future development into a more "modern" language.
His response was "You could do that and it would probably be enough to earn your PhD, since it'll take you at least three years. But I suspect you'll want to work on something else during that time".
He was right. I later learnt one of our "rival groups" attempted the same thing and it took three phd students working full time for a year to rewrite their code from fortran to C++.