Re:A Code Glitch May Have Caused Errors in More Than a Hundr...
by liability on Saturday October 12, 2019 @05:49PM
(#21236607)
'Piles-of-files' is itself a bug in an era where sqlite exists and has bindings for virtually every language. Concurrent writing can almost always be reasonably avoided by keeping the processing parallel but serializing the writes. In cases where that earnestly isn't sufficient, a 'proper' RDBMS is a good option.
Pile-of-files is 1960's era tech. We've learned much since then and our hardware is much more capable. Unless you're shooting for the retro-UNIX aesthetic for artistic reasons, it should be avoided.
HOWEVER HYPERTHREADING WASN'T ACCOUNTED FOR. Between the images_list.txt and errors_list.txt about every 20 or 30 lines the file order would be swapped. Invalidating the analysis and producing poor images.
It only took this undergrad two months to learn what was happening.