News for hackers. Stuff that matters.
Re:A Code Glitch May Have Caused Errors in More Than a Hundr... by saboot on Saturday October 12, 2019 @05:33PM (#21236511)
Oh man, a sorting error of files due to filesystem differences.. A very similar issue happened to me while doing astronomy during undergrad. This was a software pipeline for a specific telescope and I followed the guide they provided. Essentially there were two directories, one for images, and another for error estimates. Files in both directories had the same filename. The software needed text documents of the images and errors. The guide listed to do a basic 'ls images/ > images_list.txt` `ls errors/ > errors_list.txt`.

HOWEVER HYPERTHREADING WASN'T ACCOUNTED FOR. Between the images_list.txt and errors_list.txt about every 20 or 30 lines the file order would be swapped. Invalidating the analysis and producing poor images.

It only took this undergrad two months to learn what was happening.

Re:A Code Glitch May Have Caused Errors in More Than a Hundr... by liability on Saturday October 12, 2019 @05:49PM (#21236607)
'Piles-of-files' is itself a bug in an era where sqlite exists and has bindings for virtually every language. Concurrent writing can almost always be reasonably avoided by keeping the processing parallel but serializing the writes. In cases where that earnestly isn't sufficient, a 'proper' RDBMS is a good option.

Pile-of-files is 1960's era tech. We've learned much since then and our hardware is much more capable. Unless you're shooting for the retro-UNIX aesthetic for artistic reasons, it should be avoided.

Re:A Code Glitch May Have Caused Errors in More Than a Hundr... by kardos on Saturday October 12, 2019 @09:08PM (#21237542)
I don't think this is one size fits all advice. Surely once your results are measured in TBs this will fall part.