News for hackers. Stuff that matters.
Re:Missing Covid-19 test data was caused by the ill-thought-... by avian on Monday October 05, 2020 @06:00AM (#24686176)
> Using an Excel sheet as a database

How do you know it was Excel? I see no technical details in the posted article except "some files containing positive test results exceeded the maximum file size".

Re:Missing Covid-19 test data was caused by the ill-thought-... by benaadams on Monday October 05, 2020 @06:11AM (#24686233)
https://twitter.com/MaxCRoser/status/1313046638915706880

> The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum.

Re:Missing Covid-19 test data was caused by the ill-thought-... by jiggawatts on Monday October 05, 2020 @06:19AM (#24686270)
Columns!?

You're meant to add additional records as rows! (Excel supports only 16,384 columns, but 1,048,576 rows.)

Actually, this was a pet peeve of mine: Especially in the early days of COVID, just about every official or unofficial data source was an absolute shitshow:

- Transposed data (new entries in columns)

- Pretty printed dates (instead of ISO 8601 or Excel format, or... anything even vaguely parseable by computer).

- US date formats mixed in with non-US date formats.

- Each day in a separate file, often with as little as 100 bytes per file. Thousands upon thousands of files.

- Random comments or annotations in numeric fields (preventing graphing the data).

We don't all need to be data scientists, or machine learning wizards, or quantum computing gods.

But come on. This is the most basic, data entry clerk level Excel 101 stuff. Trivial, basic stuff.

Put rows in rows.

Don't mix random notes into columns you might want to sum or graph.

Don't mix units.

Have headers.

Use a sane date format.

That is all.

</rant>

Re:Missing Covid-19 test data was caused by the ill-thought-... by hef19898 on Monday October 05, 2020 @06:40AM (#24686393)
Oh, I saw excel files that contained screen shots of other excel files over cells containing data. Comments where in text fields, if you were lucky, or again screen shots, from word.

And then the same people wondered why things based on this file didn't work.

Re:Missing Covid-19 test data was caused by the ill-thought-... by RobertKerans on Monday October 05, 2020 @12:29PM (#24689237)
They seem to have hit the row limit, not the column limit. They were using Excel to collate the data coming in from different test centres: when one of the CSVs was loaded in, the row limit was exceeded and Excel cut off the extra rows without the operator realising (I'm going off reporting by Alex Hearn at The Guardian here, and he's normally pretty good)
Re:Missing Covid-19 test data was caused by the ill-thought-... by cblconfederate on Monday October 05, 2020 @06:28AM (#24686322)
lets assume that local centers people do save them in rows, which is the most logical thing to do. But probably the software where they copy/paste them requires the data in columns, so maybe they transposed the data without realizing that it was truncated to max size.

Or else it wouldn't explain why it took so many days to realize that they couldn't add more columns.

Re:Missing Covid-19 test data was caused by the ill-thought-... by sails on Monday October 05, 2020 @06:50AM (#24686441)
A question, was it possibly a csv?

ie a csv is equivalent to Excel at the Daily Mail, so they printed Excel as the issue due to it being more familiar to their readership.

(Can't think of what the technical issue would be in the case of a csv, also doubt the credibility of the source!)

Re:Missing Covid-19 test data was caused by the ill-thought-... by gridlockd on Monday October 05, 2020 @02:06PM (#24690288)
The story I heard was that it was indeed a CSV and that Excel (silently) drops anything beyond 1 million rows.
Re:Missing Covid-19 test data was caused by the ill-thought-... by shadowgovt on Monday October 05, 2020 @02:52PM (#24690757)
More specifically: they were detouring through the old XLS file format, which has a 65k row limit.
Re:Missing Covid-19 test data was caused by the ill-thought-... by mellosouls on Monday October 05, 2020 @06:39AM (#24686381)
Actually from The Daily Mail, and not clear where they are getting it from.
Re:Missing Covid-19 test data was caused by the ill-thought-... by yodelshady on Monday October 05, 2020 @06:41AM (#24686399)
Worth noting the primary source for that claim is the Daily Mail.
Re:Missing Covid-19 test data was caused by the ill-thought-... by mellosouls on Monday October 05, 2020 @06:38AM (#24686375)
I noticed this as well, and started worrying the various news items that were citing Excel since had been reading HN!

I still haven't seen an authoritative source, or one that predates that comment.

But it definitely sounds possible.