Re:Curse of dimensionality
by sageikosa on Wednesday May 30, 2012 @09:39PM
(#4045866)
I seem to remember F-tests or something like that when I did multi-dimensional variant analysis as part of my statistical geography courses back in the day.
Even though you may have data along 7 or 8 dimensions, you may be able to develop a statistcal model that captures 95% predictive value using only 3 or 4 dimensions (but the 3 or 4 dimensions do not co-relate perectly to any of the 7 or 8 dimensions you've collected), instead each predictive dimension is based on part of each of the others.
Or if I summarize: the underlying (statistically useful) dimensionality of a dataset can be smaller than the number of variables collected as many of the variables co-relate with each other.
Even though you may have data along 7 or 8 dimensions, you may be able to develop a statistcal model that captures 95% predictive value using only 3 or 4 dimensions (but the 3 or 4 dimensions do not co-relate perectly to any of the 7 or 8 dimensions you've collected), instead each predictive dimension is based on part of each of the others.
Or if I summarize: the underlying (statistically useful) dimensionality of a dataset can be smaller than the number of variables collected as many of the variables co-relate with each other.