Re:Ask YC: Your Favorite Hack
by rglullis on Thursday October 18, 2007 @01:17AM
(#69527)
Hum, good one. I have a cool non-hack, actually.
In college, I worked for a while in the applied physics department. It was an undergrad research project, where I was supposed to work with parallel systems and cluster architectures. They use software that calculates "density functional theory total-energy". Part of my project was to make this piece of software run in multiple machines and speed up the simulations. That used to take days to complete, sometimes weeks, and they wanted to reduce the period of the cycle.
I never got to know the whole theory behind the software (I am a Computer Engineer) but I did get to take a look at the manual explaining some of the algorithms, while I was learning more about parallel systems using message passing libraries.
I was just getting started on the whole "implement map, reduce, filter, barrier" using MPI (in Fortran!) when I found a description of the software where it showed how each atom in the strutcture, IIRC, needed to have its plane-wave basis-set calculated. This calculation required a long list of functions and was called for _every_ atom in the structure. Also, the software consisted of long runs of this call, getting more approximate results at each run.
Basically, I found an unbelievably parallelizable (is that a word?) piece of code in a application that was mostly serial.
Let's just say this: it took me more time to find the place in the code that have the call and even more time to make changes in the Makefile in order to use Intel Fortran Compiler than the time it took me to add 5 lines of MPI code and get the problem solved.
When we saw the speedup was directly proportional to the size of the cluster (if we wanted double of the performance, just add the double of computers), I realized that the feat (changing significantly the behavior of an program with so little modification) would be hard to repeat.
It was no "major hack", but it made me value (a) proper documentation and (b) getting a complete view of the system that I am supposed to work on and modify.
In college, I worked for a while in the applied physics department. It was an undergrad research project, where I was supposed to work with parallel systems and cluster architectures. They use software that calculates "density functional theory total-energy". Part of my project was to make this piece of software run in multiple machines and speed up the simulations. That used to take days to complete, sometimes weeks, and they wanted to reduce the period of the cycle.
I never got to know the whole theory behind the software (I am a Computer Engineer) but I did get to take a look at the manual explaining some of the algorithms, while I was learning more about parallel systems using message passing libraries.
I was just getting started on the whole "implement map, reduce, filter, barrier" using MPI (in Fortran!) when I found a description of the software where it showed how each atom in the strutcture, IIRC, needed to have its plane-wave basis-set calculated. This calculation required a long list of functions and was called for _every_ atom in the structure. Also, the software consisted of long runs of this call, getting more approximate results at each run.
Basically, I found an unbelievably parallelizable (is that a word?) piece of code in a application that was mostly serial.
Let's just say this: it took me more time to find the place in the code that have the call and even more time to make changes in the Makefile in order to use Intel Fortran Compiler than the time it took me to add 5 lines of MPI code and get the problem solved.
When we saw the speedup was directly proportional to the size of the cluster (if we wanted double of the performance, just add the double of computers), I realized that the feat (changing significantly the behavior of an program with so little modification) would be hard to repeat.
It was no "major hack", but it made me value (a) proper documentation and (b) getting a complete view of the system that I am supposed to work on and modify.