Re:The Parable of Google Flu: Traps in Big Data Analysis [pdf]
by colin_mccabe on Monday March 24, 2014 @02:20AM
(#7457000)
I guess I'm missing some context here. Why don't sales of cold medicine impress you? Why are the CDC's numbers a gold standard?
Re:The Parable of Google Flu: Traps in Big Data Analysis [pdf]
by Fomite on Monday March 24, 2014 @11:19AM
(#7458679)
Things like the sales of cold medicine have been used before, and generally speaking they're pretty poor at detecting anything. They can occasionally work well if you know something happened already, and are looking to see when it started, but as any sort of surveillance system, they haven't panned out despite some very intensive research.
The CDC numbers (which are part of ILINet) are pretty decent, and one of the things they do, which hasn't been mentioned here, is they're not trying to find all cases of influenza. They have a network of physicians in all 50 states reporting "Influenza-like Illness". These people then also send samples in for laboratory confirmation of virus type, drug resistance, etc. The numbers are also backwards revised as more data comes in, meaning they improve over the course of the year.
It's a very good system for getting an idea of how "bad" the flu season is - the main problem is that it's slow. The ideal, and what people were hoping from Google Flu Trends (and other systems like it, it's not alone) is "ILINet, but in real time".
Things like "sales of cold medicine" and the like fall under the heading of "syndromic surveillance", and believe me, they've been tried for the better part of a decade without yielding much that's impressive. Which is frustrating, because theoretically they should be amazing, but they just aren't, especially when asked to predict rather than back fit.