That assumption is not necessary, as I mentioned and mlthoughts mentions above, we do have tools to deal with discrepancies between test and train distribution.
BTW I want talking about research but meat and potatoes applications.
I think it is intellectually lazy to raise questions that even the person asking the question and everyone else knows that there cannot be an answer, especially when we dont define things (example 'real world'). Its easy-peasy to ask for things that are epistemologically impossible. On the other hand working within the limitations of impossibility theorems, under assumptions that seems reasonable, to obtain theoretical results as well as practical algorithms to solve real problems -- now that is a valuable proposition.
> [1] I'm on the second year of a machine learning PhD
Goes without saying. Those who have been through that, can spot our old enlightened newbie self from a mile away :)
No, I believe that real progress is definitely possible, "epistemologically". That is how science advances- by solving one "impossible" problem after the other. Of course, that's not how the industry rolls. In industry, if you have something that makes money, you sell it, whether it works or not and you don't bother with silly stuff like theoretical justifications and whatnot. See Waymo's driverless cars (with drivers) and Tesla's Autopilot (that can't auto-pilot) etc.
As to what is the real world- "everything that isn't in your data set" will do. My earlier comment was just me being gregarious.
And this:
>> Goes without saying. Those who have been through that, can spot our old enlightened newbie self from a mile away :)
-is distasteful. I was having a conversation and you were having a pissing contest. My bad, I guess.
No, I'm not a newbie, you are not me but more experienced and I'm probably older than you. Wrong assumptions. Again.
You might not agree, but what you demonstrate is very common among new PhD students (and I would argue that is a desirable trait). I sure had that in my days of yore.
Just clarifying that by 'impossible' I meant a mathematical impossibility theorem, not some notion of "that seems very implausible".
Mathematical science does not progress by making impossible statements possible. Rather they work by scoping out and relaxing things (conditions, assumptions and axioms).
For example, one does not deal with Russel's paradox by removing the paradox, but by scoping things out (in this case putting restrictions on how you define your sets) so that the paradox does not arise in the restricted, yet practically useful setting. OTOH other hand if someone keeps bringing up 'whoa Russel's paradox, whoa Russel's paradox that's a limitation' (albeit a well known and obvious one) that qualifies as newbie behavior, although an enlightened one.
Inference in an arbitrary graphical model is NP hard. It would be lazy to come up with a corner case where a inference algorithm with a short running time will fail. We make progress by accepting that for arbitrary graphs inference is a hard problem and then asking what is a practically relevant subset of graphs where one can indeed give an efficient solution. Pointing out obvious failure cases is exactly what you said -- a [lazy and vapid] pissing contest. It does no one any good, unless the failure case brought up is of a new and an unknown type that provides new insight. If there was any new insight in the cases you brought up I definitely missed them.
To clarify, when I say "the (entire) dataset" I mean all the data that one has at the start of a research project. What I call the "training set" and the "test set" are partitions of that dataset, used to train and validate a model. The partitions might be chosen once, or the entire dataset may be split randomly in training and test partitions during cross-validation, etc.
Now, what I say above is that the distribution of the dataset is known - you can count the instances in it, etc. What is not known is the distribution of data in the real world, where the dataset was sampled from. Most of the time, there is no way to tell whether a dataset is representative of the real world. Accordingly, there is no guarantee that an accurate model of the dataset is an accurate model of the real world.
What's more, most of the time you can't measure the performance of a model in the real world- because you don't know the ground truth. Again, you can estimate- but you can't calculate.
I think, by "test set" you may mean the real world. In that case, I don't disagree with what you say- assuming that a dataset is representative of the real world has "served us well". But, who are "we" and what is our purpose? If we are talking about machine learning research [1] then yes, that assumption has served us well: we can submit papers showing results on specific datasets and have them accepted, because everyone understands that this is the best anyone can do. And it would be tedious and pointless to accompany every result with a disclaimer about PAC learning assumptions- "this result is only meaningful if we accept that our dataset came from the same distribution as the real world" etc.
However, my concern is that all this only serves to obscure the fact that we can't really tell how much real progress we are making, as a field. I am going to side with the Great Satan, Noam Chomsky, on this one- our progress until now can only be considered progress if we accept the definition of progress that we have chosen ourselves, that of beating benchmarks and solving datasets, both chosen completely arbitrarily and with no guarantee that they have anything to do with, again, the real world.
Now we can have a big, post-modern fight about what is this "real world" I speak of, if you like :)
___________
[1] I'm on the second year of a machine learning PhD, though I should say it's symbolic, logic-based machine learning i.e. Inductive Logic Programming- which perhaps accounts for my preoccupation with subjects the mainstream of machine learning research today probably finds irrelevant.