Re:Power of Small Optimizations
by chriswarbo on Saturday February 10, 2024 @12:24PM
(#39328148)
Urgh, reminds me of a production error that was caused by a job running on a timer, assuming that some other job would have finished by then. That lead to two obvious questions:
- Why is it on a timer, instead of being triggered when the first job was done?
- Why was that first job taking over an hour to process XML data?
Turns out the second one was accidentally-quadratic, since it was parsing a whole document to process its first record; then re-parsing the whole document to process its second record; etc. and this made it very slow when the XML was reaching several hundred MB.
Unfortunately, the solution we went with was to set the timer to run 20 minutes later...
- Why is it on a timer, instead of being triggered when the first job was done?
- Why was that first job taking over an hour to process XML data?
Turns out the second one was accidentally-quadratic, since it was parsing a whole document to process its first record; then re-parsing the whole document to process its second record; etc. and this made it very slow when the XML was reaching several hundred MB.
Unfortunately, the solution we went with was to set the timer to run 20 minutes later...