In reality (as you describe), these are completely arbitrary human-designed heuristic scores with most likely no statistical significance.
I really wish we could qualify these "rankings" with a more honest term , like:
"statistically useless, arbitrarily rated average of multiple human designed score scales, meant to loosely relate to some quality we want to measure, but in reality is more a game of politics and adversarial score optimization."
But that doesn't have the same 'ring' to it as "top country rankings in innovation".
So its an equally weighted average of 7 categories. The data was reported by the nations themselves.
Now the key takeaway is that the US dropped out of the top 10. Comparatively, you can tell something is changing in the US causing a drop.
Metrics are just indicator, and can sometimes misrepresent the reality, but more often, there's truth in the metrics also. Its hard to say what the impact of this innovation score is, is it economic, or is it social, but clearly the score change is due to real realities changing.
The reported numbers from other countries, or the reported numbers from the US, could also be flawed. Just because numbers have changed relative to one another, doesn't mean that countries have done so as well. You are assuming a particular causal relationship when there are several plausible alternative theories available.
> more often, there's truth in the metrics
Metric isn't the right word. It would make sense for tangible qualities like area, population, and even GNP. But the measured quality here is "innovation," with an amalgam of other characteristics taken as the one true proxy thereof. It's a ranking, but it's not actually measuring innovation.
> clearly the score change is due to real realities changing.
It could also be a change in reported numbers. Or even random noise that tends to revert. These numbers may not even be relevant to the true seeds of innovation.
It's the change in relative rankings that is the useful data from these rankings, rather than the absolute ranking.
> Comparatively, you can tell something is changing in the US causing a drop.
Yes, one of the factors they used to come up with this rating. Now - the question is, are those factors actually correlated with innovation?
Now, maybe the US isn't as innovative as Sweden is. Ok. What exactly does that mean? Why do I care if the percentage of graduate educated people is higher? How does that actually affect innovation? Are those people releasing new, globally-changing products and services? What are some examples?
What does it mean if Samsung has more US patents than any other company besides IBM? Is IBM more innovative than Google?
It's fun and popular to bash the U.S. (has been for some time) but I really don't see much meaning behind these rankings. It's not an in-depth study. Amazon has more criteria for picking a HQ. Do you really think Bloomberg can look at these '7 criteria' and come up with a meaningful estimation? No.
It means exactly the same thing it meant when US was among top. There were people who were interested in it and sometimes happy about it. Those very same people are still interested, but this time wonder whether it means something is changing for worst.
Even the notion "hard science" vs "soft science" is a very slippery slope, IMO. We should not seek to speak of "hard" vs "soft" science; instead, we should focus on distinguishing good from bad science. Even science performed on incredibly difficult and complex topics (with immensely numerous confounding variables, near impossibility of controlled trials/experiments, etc.) can still be done correctly! One simply needs to adhere to statistical rigor and qualify conclusions from data with appropriate levels of uncertainty, withhold conclusions with no predictive power or statistical significance, and publish meaningful negative results just as frequently as positive results.
This movement of lax scientific rigor within fields that call themselves "sciences" is incredibly dangerous and threatens to erode the credibility of all results/fields that call themselves "scientific" in the eyes of the general public -- the vast majority of whom do not have the time, energy, or ability to review each field and/or publication to understand how rigorous and honest it actually is.
When you read about problems of anti-intellectualism and public distrust in science, the first thing we should do is look to the "sciences" (and bad journalism) that justify this distrust.
"statistically useless, arbitrarily rated average of multiple human designed score scales, meant to loosely relate to some quality we want to measure, but in reality is more a game of politics and adversarial score optimization." while honest and true doesn't get the masses clicking. Honest and true doesn't get stories bumped to the frontpage of HN.
I do feel for these journalists. They are like daily vloggers who have to deal with the constant pressure of generating content every single day to make money. That's like clickbait is so rampant in both the traditional and social media.
1) Formally define what is meant by "innovation" in terms of clearly measurable outcomes.
2) Measure this clearly-defined quality among all countries and many sample points through time.
3) Try to separate out explanatory variables for the quality being measured. Build these data driven statistical models to model this formally defined "innovation" quantity -- not using hand-tuned weights of various measures, as these "rankings" or "indexes" often do.
4) Try to predict a probability distribution of the "innovation" quality, using models developed in step 3.
Step 1 should be qualified with explanation that this human-designed definition is an imperfect, and that all results should be understood in the context of this formal definition.
Step 2 should be qualified with notes of any possible limitations in the sampling methodology (availability of data, etc.) and how this factors into error margins.
Step 3 should be qualified with sufficient explanation that it's a model of reality derived from data, and therefore risks overfitting/underfitting/etc. errors.
Step 4 should be qualified with an explanation that this is a prediction based on the above model fit, and therefore is subject to potential errors compounded by any of the previous steps.
That would be the scientifically/statistically responsible and rigorous thing to do. But I suppose I'm crazy to expect Bloomberg to aim for any level of rigor in these "indexes".
Surely you can see that this is not possible, and any attempted would be superficial and would be gamed anyway?
It’s not feasible for a human to define such a metric formally upon ‘environmental variables’ (such as education stats, graduation rates, etc.) — quite obviously, as you say — yet trivial to define it as an “outcome measure”, where we directly measure the quantity in question (no matter how difficult or expensive to sample this variable).
To the “quality of living index” example: One could design a polling methodology to fairly reliably gauge people’s overall happiness and satisfaction in a country. This polling would be expensive, so we couldn’t do it super broadly or super frequently — and that’s why we use the subsequent steps described in my parent post (on forming statistical models to separate out connected variables that we can easily and cheaply measure to approximately model the “ground truth” happiness metric).
You can then use this “model fit” to predict this extremely expensive “ground truth” notion of individual happiness in this case, on a much more frequent and granular basis than would ordinarily have been feasible using a ground truth gathering method like a polling process.
The innovation you can see and experience in a product like Facebook or VKontakte is just the tip of the iceberg when it comes to the unseen tech from the three-letter agencies of many different countries, that are mining, scraping, and reducing the data.
But there's a lot of anti-innovation going on in that sector too, I'm sure.
Of course all that means is that it isn't a perfect proxy. It may well still be a good proxy. Tough to call.
1. Gross tertiary enrollment ratio
2. Percentage of working-age population with advanced level of education
3. Annual new science and engineering graduates as a percentage total tertiary graduates
4. Annual new science and engineering graduates as a percentage of the labor force
huh?
luckily my friends and I are black-shifting these deep blue pigs, who eat people, and are creating a new era where people are allowed to write their own story.
Using the Bloomberg list, Austria ranks 12th overall, but 5th on R&D. Their figure is dragged down by a 26th rank in High-Tech Density. Then, Germany's High-Tech density is ranked 3rd overall. Perhaps one could hypothesise that much of Austria's R&D effort ends up commercialised in Germany.
Ranking countries is a dodgy business, even more than ranking colleges. A different set of weights or ways of measuring things could give you totally different answers.
If you're looking for a way to claim the rankings are biased, you might argue that this up-ranks countries that value credentials over actual innovation. Or you might claim that these days, an undergrad education is enough to go out in the world and innovate and that countries that send more students through grad school are wasting their time. Or you might claim that the US is a developed country with a developing country attached, which drags down the averages. And probably California, NY, MA and a few other states considered independently would rank highly.