Missing Covid-19 test data was caused by the ill-thought-out use of Excel

Re:Missing Covid-19 test data was caused by the ill-thought-... by jononor on Monday October 05, 2020 @03:16PM (#24690999)

Always use checked pre- and post-conditions in a data pipeline.

This simple post-condition would have caught this issue: The sheet after merge operation must have a number of rows equal to the sum of number of rows for all merged sheets.

Assuming this is a merge of a standardized input, then another post-condition might be: The number of columns in output shall equal the number of columns in the input. Might want to check header names, and order as well.

Thinking in terms of universal properties, and putting the checks into production, is better than unit-testing.

[ Parent | on HN ]

Re:Missing Covid-19 test data was caused by the ill-thought-... by 1_player on Monday October 05, 2020 @05:08PM (#24692072)

> Always use checked pre- and post-conditions in a data pipeline. [A] simple post-condition would have caught this issue

I'm not sure they teach that in medical school.

[ Parent | on HN ]

Re:Missing Covid-19 test data was caused by the ill-thought-... by avibhu on Monday October 05, 2020 @10:15PM (#24694162)

Which I feel might be indicative of a larger problem. If you are going to deal with data (and its processing), it only makes sense to teach those skills in school itself.

[ Parent | on HN ]

Re:Missing Covid-19 test data was caused by the ill-thought-... by swiley on Monday October 05, 2020 @10:27PM (#24694223)

They spend so much time memorizing random trivia maybe a course in handling data wouldn't be a bad idea.

[ Parent | on HN ]

Re:Missing Covid-19 test data was caused by the ill-thought-... by ArnoVW on Tuesday October 06, 2020 @04:11AM (#24695652)

Give them a break. They don't have the luxury of a highly-structured field, purposefuly constructed by humans (engineers & mathematiciens on top of that) with only 50 years of legacy.

They have to deal with a field that was haphasardly constructed by nature over the course of 4 billion years.

Imagine having to retro-document a code base with 4 billion years of history, created solely by junior developers fresh out of college..

[ Parent | on HN ]

Re:Missing Covid-19 test data was caused by the ill-thought-... by jussij on Tuesday October 06, 2020 @03:47AM (#24695555)

This quote from the article says this work was being done be a developer(s) and if that is the case they should have know better.

The problem is that PHE's own developers picked an old file format to do this - known as XLS.

[ Parent | on HN ]

Re:Missing Covid-19 test data was caused by the ill-thought-... by Closi on Tuesday October 06, 2020 @05:41AM (#24696108)

It's unlikely this was done by a developer in reality, and it's more likely to be a data analyst that was called a developer in the press release.

[ Parent | on HN ]

Re:Missing Covid-19 test data was caused by the ill-thought-... by jen729w on Monday October 05, 2020 @04:28PM (#24691652)

Even just within my own sheet I always build in some sort of idiot-check, the assumption being that I’m the idiot.

Do a SUM and match it against a COUNTIF on another column, or something. It doesn’t really matter what it is but if the data is important at all, I always scatter little checks throughout.

The case described in this article sounds like someone who just doesn’t know how to use Excel. I mean why in the name of VLOOKUP would you override the default and choose to save as .xls? That was a conscious decision. Anyone worth their salt knows that’s stupid.

Excel is not to blame here.

[ Parent | on HN ]

Re:Missing Covid-19 test data was caused by the ill-thought-... by dkarp on Monday October 05, 2020 @05:49PM (#24692427)

I’d guess they wrote a script to merge the csv file and export in xls. Xls might have been the default of the library they used in a rush. I think this may be the case with some python libraries that generate excel files as xlsx is a more complicated format.

[ Parent | on HN ]