There’s a lot of folks out there though who learned the mechanics of linear regression in a bootcamp or something without gaining an appreciation for the underlying theories, and those folks are looking for low p-value and as long as they get it it’s good enough.
I saw this link yesterday and could barely believe it, but I guess these folks really live among us.
https://stats.stackexchange.com/questions/185507/what-happen...
For example, if your phenomenon is observable at 50 Hz, maybe even 10 Hz, then any higher temporal resolution does not give you new information, because any two adjacent datapoints in the time-series are extremely correlated. Going the other way, at a very low sampling frequency you'd just get the mean, which might not reveal anything of interest.
If you bin 100 Hz data at 50 Hz, are they the same? Is the Fourier spectrum the same? If you have samples of different resolution you must choose the lowest common denominator for a fair statistical comparison. Otherwise, a recording between a potato and an advanced instrument would always be "statistically different", which doesn't make sense.
If you don't find "anything", the old adage goes "the absence of evidence is not the evidence of absence", so statistics don't really fail here. You can only conclude that your method is not sensitive enough.
For example, I’ve encountered the belief that just by recording something at ultra high temporal resolution gives you “millions of datapoints”. This then has all sorts of effects on the breakdown of statistics and hypothesis testing (seemingly).
In reality, the replicability of the entire setup, the day it was performed, the person doing it, etc. means the n for the day is probably closer to 1. So to ensure replicability you’d have to at least do it on separate days, with separately prepared samples. Otherwise, how can you eliminate the chance that your ultra finicky sample just happened to vibe with that day’s temperature and humidity?
But they don’t teach you in statistics what exactly “n” means, probably because a hundred years ago it was much more literal in nature. 100 samples is because I counted 100 mice, 100 peas, or 100 surveys.