News for hackers. Stuff that matters.

A comment on Missing Covid-19 test data was caused by the ill-thought-out use of Excel.

Re:Missing Covid-19 test data was caused by the ill-thought-... by rossdavidh on Monday October 05, 2020 @06:32PM (#24692782)
While I am not in the habit of defending Microsoft, I think it should be pointed out that it wasn't using "Excel" that was the biggest problem, but using "13-years obsolete format for Excel". Not that there was any reason to be using Excel for this, but it's a bit unfair to blame Microsoft for something they never claimed XLS could do, and provided a much better capacity in XLSX 13 years ago.

Again, it's hard to cry too many tears for Microsoft, but it does seem a bit off-target to blame "Excel" for this...

[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by bobbylarrybobby on Monday October 05, 2020 @09:08PM (#24693839)
The issue with is excel is that its failure mode is suboptimal. Programming languages will generally crash at runtime when an unrecoverable error occurs (index out of bounds, memory allocation failed, null pointer, etc). But excel just throws away your data?
[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by curiousllama on Monday October 05, 2020 @09:41PM (#24694002)
If it fails, it’s often silent. If not silent, hidden. If not hidden, misleading, if not misleading, opaque. If not opaque, trivial.

Excel is a wonderfully powerful tool that’s very bad at handling errors clearly.

[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by lbruder on Tuesday October 06, 2020 @03:12AM (#24695389)
It's the PHP of business software. Powers half the market, but boy do mistakes cost you.
[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by tomc1985 on Monday October 05, 2020 @09:23PM (#24693905)
Indeed, you have to manually inspect results to verify correctness. If you're lucky maybe your error will result in one of those weird error values being shown instead, but way too often that is not the case

Not to mention that complex formulae are still usually expressed as a bunch of gobbledygook in the cell value textbox, which is about as easy to parse as minified Javascript. And that's to technical users like ourselves.

[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by logicOnly on Monday October 05, 2020 @09:56PM (#24694082)
I had Excel error out when I tried to open 10 million rows.
[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by sueders101 on Monday October 05, 2020 @10:10PM (#24694135)
Excel tops out a little more over a million rows without specifically handling of larger files. I’m not sure excusing it, just a heads up to all the junior analysts out there who haven’t dealt with the problem(not implying you’re a junior analyst, just wanted to point something out to those who aren’t familiar with the issue).
[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by logicOnly on Tuesday October 06, 2020 @07:53AM (#24696742)
No worries, Excel is nice for small/medium sized data and has the UI/Visualization which is nice.

The story I mentioned was because I wanted to look at the data before I started parsing it. I had full expectations to use either sqlite or Pandas.

[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by hurrrrrrrr on Tuesday October 06, 2020 @07:05AM (#24696491)
> Programming languages will generally crash at runtime when an unrecoverable error occurs (index out of bounds, memory allocation failed, null pointer, etc).

JS would like to have a word with you.

[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by Closi on Tuesday October 06, 2020 @04:27AM (#24695737)
In this case, Excel throws an error and says that not all rows have been loaded.

It seems that warning has just been ignored by the user.

[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by kelnos on Monday October 05, 2020 @09:04PM (#24693822)
Sure, and I think at least the headline ("ill-thought-out use of Excel") avoids blaming MS or Excel itself.

Ultimately tools are built for particular things, and if you choose to use a tool for something it's not built for, and it breaks catastrophically, that's on you.

[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by sriram_malhar on Monday October 05, 2020 @10:52PM (#24694325)
No. Excel is definitely to blame. A simple error "cannot import/export more than 65k rows" would have done the trick. Not silently throw away rows.
[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by corty on Tuesday October 06, 2020 @02:40AM (#24695261)
The premise of excel is very much "don't bug me, just do it". The highly praised UI simplifies away corner cases and errors intentionally. Summing over an empty cell? Most probably zero, why throw an error and confuse the user? The simplified UI makes correctness hard to impossible, so excel is just unsuitable for tasks where correctness is desired.
[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by account42 on Tuesday October 06, 2020 @04:49AM (#24695856)
Interpreting an empty cell as 0 is making an (often correct) assumption about what the user wanted.

Throwing away data without warning is almost certainly never what the user wanted.

[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by ComodoHacker on Tuesday October 06, 2020 @02:26AM (#24695212)
You get CSV files as input and have to upload them into the database. You don't need Excel at all, neither in XSL, nor in XSLX format.

My bet is the biggest problem here is subcontracting this work to the lowest bidder, presumably from some developing country.

[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by baud147258 on Tuesday October 06, 2020 @09:57AM (#24697672)
> subcontracting this work to the lowest bidder

knowing a little how things (don't) work in the UK, it's likely subcontracted, but to a company belonging to a mate of the director in charge of the whole thing

[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by yawaramin on Tuesday October 06, 2020 @01:12PM (#24700106)
I'd take that bet ;-) From the article:

> > And it appears that Public Health England (PHE) was to blame, rather than a third-party contractor.

[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by nabilhat on Monday October 05, 2020 @09:22PM (#24693902)
Over-13-years old Excel already opened delimited flat files natively, and it threw warnings when you opened a .csv file that exceeded the worksheet row limit. Someone in the distant past was just clever enough to hack together the script (with no error handling) without being curious enough to know it wasn't necessary in the first place.
[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by mcfedr on Tuesday October 06, 2020 @02:24AM (#24695203)
If they had used the newer format, we'd be talking about this in a few weeks time when the larger limit was reached.

This is a problem of bad coding, and using the wrong tool for the job.

A defensive coding practice would have prevented this from going unseen. Using a database to store data would have prevented such arbitrary limits.

[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by rossdavidh on Tuesday October 06, 2020 @09:39AM (#24697539)
Yeah, something like "Missing Covid-19 test data was caused by using spreadsheet instead of database", would be more accurate.
[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by yawaramin on Tuesday October 06, 2020 @12:58AM (#24694831)
Nobody is blaming Microsoft. From the article:

> And it appears that Public Health England (PHE) was to blame, rather than a third-party contractor.

[ Parent | on HN ]
Re:Missing Covid-19 test data was caused by the ill-thought-... by tonyedgecombe on Tuesday October 06, 2020 @03:53AM (#24695585)
Blaming PHE seems to have become a regular occurance this year. I wonder if it's justified or not.
[ Parent | on HN ]