If it helps anyone, the FiveThirtyEight article describes a scenario where people take a survey about immigration status and voting. Most legal citizens will correctly identify themselves, but some will accidentally check the wrong box and say they are an illegal immigrant. If you have a billion citizens and 10 illegal immigrants truly taking the survey, and people check the wrong box 1 in 1000 times, your "percentage of illegal immigrants who vote" statistic will be about the same as for citizens (because almost all reported illegals will be citizens). Collecting more data won't help.

It's a very good article, though in the context of deciding how many variables should be in a model of some complex phenomenon, this example is a little tougher to wrap your head around. It's not quite a predictive model, but there were some variables left out. A naive model I suppose is "this data is generated by infallible respondents", whereas a better model would incorporate that error rate. There isn't as much of a question about which pieces of information are relevant, though, like you might encounter when trying to predict future drug use from household income, race, age, number of books read as a child, number of pets, and so on.

The other common example of the same phenomena is a test for a deadly genetic defect with a 1% false positive rate. If the incidence of the defect is .01% and you test positive, its actually more likely that you don't have the disease. (although this can be solved with bayesianism over frequentism).

This is a pernicious misunderstanding of "frequentism", often found among people who studied statistics mainly by reading comics.

http://web.archive.org/web/20130117080920/http://andrewgelma...

That's incredibly unnecessary. My understanding of statistics is not derived from comics (and the first time I heard that example was in a statistics course), and the link you post doesn't actually address what I stated. It addresses an actual mistake in the comic, which is a mistake that I didn't make.

Here's Andrew, the author of that blog post:

> Yes, I think it makes a lot of sense to criticize particular frequentist or Bayesian methods rather than to criticize freq or Bayes statisticians.

Which is exactly what I did. There are times when frequentist methods are effective. I just wouldn't use them to tell me that I have a disease.

Doesn't this make the assumption that "illegal immigrants" won't check the wrong box, intentionally or by accident?

citizens labeled illegals: 1,000,000

illegals labeled citizens: 0.01

Illegal immigrants have no business being anywhere near that particular checkbox.

If you could find a link to the FiveThirtyEight article I'd really appreciate it. Thanks.

It's linked in the grandparent's comment. https://fivethirtyeight.com/features/trump-noncitizen-voters...

I think the key point is that in the case of sampling, you always assume that your signal has finite bandwidth, so you can claim to avoid aliasing so long as you sample suitably often. Further, the time duration over which one makes measurements puts a cutoff on the frequency resolution. Both of these essentially serve the purpose of implicit model regularization (aka bias).

For sufficiently well-behaved signals, the estimator of the strength of various frequency components (i.e. the Fourier transform) is pretty stable as one enlarges the window of measurement, which is post-hoc validation that the signal is well-behaved.

This might not be true for very weird signals, and enlarging the window of measurement might significantly change model parameters--meaning that one might need a non-parametric model to get by (enlarging number of parameters with number of measurements) rather than restricting to a finite number of frequency models. Eg: Suppose the first 100 measurements show a nice sinusoid of some frequency, but the next 100 measurements show a pretty much flat signal. Then, you are forced to revise downward the parameter corresponding to the importance of the sinusoid component, and increase the importance of the zero frequency component. The thing is, one never truly knows how the signal is going to behave in the future. Any parametric model is a bias that things won't get too complicated.

Fundamentally this is related to the induction fallacy. The doge meme image actually illustrates it pretty well.

https://en.wikipedia.org/wiki/Problem_of_induction

Another closely related thing is the No Free Lunch Theorem.

https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_op...

These are concepts I believe are very important to internalize if you work with machine learning. Fundamentally we are making predictions (ie. guessing) on the nature of entirely unknown information. So there is a certain inherent impossibility to the task in the general sense. It shouldn't always work.

The irreducible error is unrelated to the bias/variance trade-off. It's part of the overall error of course, but the bias/variance error is in addition to that. Unless I am misunderstanding your point.

Sorry, I did not mean to be that they were the same. And, to that point, I was greatly projecting based on my beliefs that have grown. The number one thing I keep having to re-stress and learn is that these are probabilistic models.

So, if anything, I only meant they were related in that they are both good targets to internalize when working with ML. If there are better targets, I'm definitely interested in learning more.

Ultimately, everything is a probabilistic model. It'd be ridiculously impractical to try to incorporate all of one's knowledge into an analytic model. Instead, we generalize as appropriate and encapsulate the rest as a random variable, sometimes called "error". Plus, God plays dice and all that.