https://en.wikipedia.org/wiki/Lucas_critique
and also https://en.wikipedia.org/wiki/Campbell%27s_law
https://en.wikipedia.org/wiki/Goodhart%27s_law
Edit: I think this is not the first time the good people from lesswrong dug up some well known idea and gave it a new name. Good thing, too, giving this important concept more attention. Too often we forget how many people have dealt with the problem of modeling complex systems in the past. And while we can not read everything, it's often a good idea to have at least a glance at where they failed!
Too often I read/review some new "revolutionary" paper based on the idea that hey, we can model this process (involving people) like XYZ from physics, where this stuff works great! Surely, this is better than the plebian approaches in the literature! And then, to the shock of all involved, it doesn't work great....
Also: https://xkcd.com/793/
It's not that. It's that when the system you model responds to the existence of your model, it becomes anti-inductive. It's no longer like weather, but is now like the stock market[0]. Your model suddenly can't predict the system anymore[2], it can at best determine its operating envelope by estimating the degree to which the system can react to the existence of the model.
--
[0] - I use the term anti-inductive per LessWrong nomenclature[1], but I've also been reading "Sapiens" by Yuval Noah Harari, and there he uses terms "first order chaotic" for systems like weather, and "second order chaotic" for systems like the stock market.
[1] - Introduced in https://www.lesswrong.com/posts/h24JGbmweNpWZfBkM/markets-ar....
[2] - I think it becomes uncomputable in Turing sense, but I'm not smart enough to reduce this to the Halting Problem.