Still... in particle physics, for example, I've heard that there was consensus in favor of certain particles existing, even to the extent of "multiple experiments have achieved an N-sigma result showing that this particle exists," for some fairly high N. And then a later experiment showed that the particle simply didn't exist, and that the previous results were experimental error. (Specifically, experimental bias – they were expecting to see certain results, and so they were probably subconsciously combing the data for such results.)
My knowledge on that subject comes from Veritasium, not from reading physics papers or following along with experimental breakthroughs. But I've seen similar things happen in ML, where people are pretty convinced a certain technique is good (hello, style mixing in StyleGAN) and no one ever bothers to check whether it's true in reality (style mixing is what prevents StyleGAN from being as cohesive as BigGAN).
I guess I broadly agree with your point. There is consensus that black holes exist, and that we've even observed one, so it's pretty likely that if you were to travel there, you'd find a black hole. But given how much science itself has changed throughout history, alarm bells start going off when hearing things like "Consensus is extremely common, and it's almost universally a good thing."
Take your example about StyleGAN vs BigGan, I assume once it became clear that the latter was superior to the former that likely resulted in changes to existing architectures that then found additional improvements. This change in consensus is what enables that and is a good thing.
I think the problem here is conflating consensus with faith, that is assuming truth in absence of data. Consensus is easily changed with the introduction of new data, faith hangs on no matter how much evidence is put forward that it's horseshit.
Well, no. :) But I think the "no" is because of the uniqueness of ML rather than a "no" to your point in general. You might be right about other fields; I don't have experience there.
In ML, there an enormous number of techniques. Style mixing was presented as a core feature of StyleGAN (https://arxiv.org/abs/1812.04948) and was enabled by default in the codebase (https://github.com/NVlabs/stylegan/blob/03563d18a0cf8d67d897...).
So there's a lot of "inertia" -- for example, when StyleGAN 2 came out, style mixing was still the default (https://github.com/NVlabs/stylegan2/blob/f2f751cdc7f996e3138...).
I haven't had time to dig into StyleGAN 3, but I suspect that style mixing might still be enabled by default.
It wasn't until we did a detailed, methodical analysis side-by-side with BigGAN, specifically to answer the question "Why is BigGAN so much better for diverse datasets?" that, on a whim, I turned off style mixing and was astonished to see BigGAN type quality pop out of a StyleGAN type arch.
Discoveries like that usually go unnoticed, frankly because it's a lot of effort to write a paper specifically to say "Hey, if you're training StyleGAN, definitely turn off style mixing. It only seems to work well on faces."
However, if such a paper were to be written, and accepted into a peer-reviewed journal, then your original point would probably be valid. So I don't even know if it's worth writing all of this -- I just thought it'd be interesting to point out the "Well, not really" in this case. The knowledge ends up floating around on Twitter and Discord rather than being transmitted via scientific papers...
But, this all does tie in to your final point:
> Consensus is easily changed with the introduction of new data, faith hangs on no matter how much evidence is put forward that it's horseshit.
It's remarkably easy for old, accepted ideas to hang around. You'd think it'd just be a matter of "Run the experiment; experiment proves thing; thing becomes accepted." But in practice it's felt quite different...
The thing is, everything you're saying is true in general. As t approaches infinity, there tends to be more and more consensus about older ideas, like the existence of black holes, or the validity of laws like F=ma. So we should probably pay attention when there is 99% consensus on a particular topic.
But, for example, one reason I wouldn't want to publish a paper claiming style mixing was bad, is because it would contradict the results of Karras, who is famous. I'd better be very certain about my claim! So there's sometimes a reluctance to contradict the consensus, too, which ends up equivalent to "faith" in your example -- we have faith that famous scientists are correct. (They usually are.)
As a cherry on top, I'll just leave a link to Feynman's messenger lectures: https://www.youtube.com/watch?v=-kFOXP026eE&ab_channel=TalkR... ... the history of science is fascinating. I'd dreamt for years of becoming a scientist, but the actual experience turned out to be surprisingly different than what I thought it'd be. I love it though -- all these weird corner cases are the spice of life.
In a thread where people are parsing the meaning of consensus with respect to climate change, citing the atypical example of the state of consensus in a unique field is going to contribute to the loss of context that makes people confused about the original subject at issue in this thread.
It's great to know about BigGAN vs StyleGAN, but I'm almost positive people will skim this detour into increasingly detailed elaboration on machine learning and think the upshot is supposed to be "and therefore we shouldn't put stock in climate science consensus."
Thanks for your really in depth and informative reply though, I'm only cursorily interested in ML through its relation to data, which is my speciality. My exposure to ML is mostly through said ML teams making requests for data and systems like feature stores to run their models so I have some understanding of how they construct their overall architecture but not the specifics so this was super interesting.
Regarding your comment on ML and consensus, I think there is a bit of a misconception of what scientific consensus is. I would say it is much broader than the examples you give. ML is essentially terrible example to argue about consensus, I would say there isn't even consensus about what ML even is, let alone any of the details of how you train a network.
Regarding your earlier example about loss, I'm a bit confused. Isn't it that in optimisation theory the goal that you want to minimise your loss (so lower loss is always better)? Now that might not mean that you should travel along the steepest gradient (this might lead you to a local minimum) and your loss might not represent what you actually want to optimise for (i.e. your loss function is faulty). But from a fundamental theoretical point of view, you always want the lowest loss?
The cliff's notes are: the first time I was able to train GPT 1.5B on gwern's poetry, I was super proud of myself, because it was extremely hard to get 1.5B to train back in those days. So we let it train as long as possible, and the loss kept dropping lower and lower. So we'd periodically dump samples -- surely we'd get better poetry, right?
No. Eventually the model degenerated into broken-sounding English. Because our training data was 50MB, so it quickly forgot how to speak English at all -- the prior knowledge got wiped out, and was replaced by that 50MB dataset.
The term for this is "overfitting," and it's remarkably easy to run into it in practice. The cure is to have more data. But how do you gather more than 50MB of poetry? That's a lot of poetry. So what do you do when you want to train a poetry model?
The answer is to train a general model on a massive dataset (https://arxiv.org/abs/2101.00027), and then to fine-tune it on a smaller dataset (like 50MB of poetry). But during that fine tuning step, you'd better watch out, because everything you thought you knew about ML is wrong: You need to have a gentle downwards-sloping loss curve -- steep curve is terrible, since it means you're destroying the old knowledge without giving it a broad view of the new knowledge. And you need to stop before hitting loss < 2.1, otherwise you'll be overfitting and the model will suck.
I suspect that hundreds of great ML models were overlooked for these reasons. There are hundreds of thousands of people messing around with ML models for fun. Lots of them try to train those models on small datasets, because small datasets are all you have when you start out. So they google how to do it, hear "Low loss is good," train overnight, dump samples, and then end up going "Oh well, the model seems to suck, but the loss is low, so I guess I didn't do it properly." They probably did train it properly, but they weren't continuously dumping samples, so they didn't notice they needed to stop before the model turned into word salad. (Or art salad if it's a GAN.)
I would say that from a fundamental theoretical point of view, "you always want the lowest loss" is only true when you have an infinite amount of data in a target domain. But in practice, you probably have a few gigabytes if you're lucky. Even imagenet is only 1,000 different image classes, which means it's possible for a model to memorize those 1,000 classes and overfit, rather than achieving a general "understanding" of different kinds of objects.
I got all of these details wrong when I first started out, so it was really eye-opening to see that it's a terrible idea to rely on common knowledge in ML. Common knowledge is an excellent starting point, but you quickly learn to think for yourself, or you just end up imitating what the big players are doing while wondering why none of your models seem to be working.
(... I guess I wrote a detailed reply after all. Ah well, sleep is optional as long as you choose to ignore the effects of sleep deprivation. That's how that works.)
I think with everything you describe is one of the main "gripes" I as a more traditional engineer/scientist have with ML. It often seems like a giant black box that I know is generally vastly overfitted and it is often extremely difficult/impossible why we get the solutions that we get, i.e. we don't really get scientific insight. Nevertheless I realise it is an extremely powerful tool.
Hope you can catch up with some sleep today, sleep deprivation effects can only be ignored for so long (especially as you get older as I have noticed).
The most biased super-precise physics I can think of offhand is when someone measures a physical constant slightly wrong and then subsequent results all lean in the same direction. This happens more than once, most notably with fundamental electric charge, but it's absolutely nothing compared to a nonexistent value!
https://en.m.wikipedia.org/wiki/List_of_experimental_errors_... has https://en.m.wikipedia.org/wiki/Oops-Leon? But that is a single experiment which very specifically did not have a high sigma result.
I’m not sure what your point is, unless it’s saying that a lack of consensus is indicative of being wrong.
Which would support the commenter you replied to’s argument.
FWIW, my experiences have been the precise opposite. My colleagues never seem to have consensus about anything. But companies move forward with our recommendations, both because we're more likely to be right, and if our recommendations happen to serve the company's goals.
I'm sure a bunch of people will read into this comment, but I am speaking purely about my experiences as an ML scientist. It would be valid to say that ML represents a very small fraction of the global scientific apparatus, and I certainly don't claim to know what it's like to work at CERN.
It surprised me, though, how often you were right -- there often is consensus, in the sense that everyone reads about a certain technique in papers and automatically assumes it's applicable to their own situation. For example, everyone in language modeling almost universally agrees that "lower loss is better," ("loss" is a specific ML term,"). But that's only true for general language models trained on large quantities of data. If you're fine-tuning a language model, you almost never want the steepest loss slope, because it means your learning rate is too high and you'll destroy the existing knowledge.
I would say that experience matters, and each situation is different. It's super tricky to make an argument that consensus is both universally common and universally good.