There's a lot of interest in various ML communities on more efficient training and inference. Both vision and NLP have had a growing focus on these problems in recent years.

I think you make a good observation that much of ML progress is driven by tinkering with existing models, though instead of describing it as more "alchemy than science" it's probably more accurate to say it's very experimental right now. Being very experimental is neither unscientific nor unusual in the development of knowledge. James Watt worked as an instrument maker (not a theoretician) when he invented the Watt steam engine in 1776 [1], and at the time the idea of heat as Phlogiston [2] was still more prevalent than anything that looks like modern thermodynamics. Theory and practice naturally take turns outpacing each other, which is part of why we need both.

I'd also caution against the belief that experimental work doesn't require "particularly demanding thought". There are many things one can tweak in current ML models (the search space is exponential) and, as you point out, the experiments are expensive. Having a solid understanding of the system, great intuition, and good heuristics is necessary to reliably make progress.

For those who are interested in the theory of deep learning, the community has recently made great strides on developing a mathematical understanding of neural networks. The research is still very cutting edge, but the following PDF helps introduce the topic [3].

[1]: https://en.wikipedia.org/wiki/James_Watt

[2]: https://en.wikipedia.org/wiki/Phlogiston_theory

[3]: https://www.cs.princeton.edu/courses/archive/fall19/cos597B/...

that course from princeton looks great! This paper is a nice short read and gives some geometric insight: https://arxiv.org/abs/1805.10451

Apply to TFRC! https://www.tensorflow.org/tfrc

They are very permissive. And you get to play with $500k worth of hardware. Been a member for over a year now. Jonathan is singlehandedly the best support person I've ever worked with, or perhaps ever will work with.

I would've completely agreed with you if not for TFRC. And I couldn't resist the opportunity of playing with some big metal, even if it's hard to work with.

just applied. thanks for sharing!

>We are a long way from the type of simulations done for protein folding and materials strength and basically every other scientific field.

Have you not heard of alphafold?

Thanks for mentioning Alphafold. This article by the DeepMind team is very insightful.

https://deepmind.com/blog/article/alphafold-a-solution-to-a-...

> Anyways, my point is, none of this required math or genius or particularly demanding thought. It was basically let's tinker with this until we find a way that's better, using guess and check. No equations needed.

I get that you'd like to have a clear theoretical basis for what works and we're far from there. But in the meantime we're stumbling in the dark, discovering tricks and forming intuitions, not knowing even where the road is going to lead us.

This is an evolutionary process of ideas, similar to biological evolution that managed to make us. If you know where you're going you can optimise your actions but when you don't even know what might be useful later on, then all attempts are good. They increase diversity and discover blind spots. Some of them will be the stepping stones for the future, but we can't say in advance which and how.

Link to a long discussion about the evolution of ML ideas and the book "Why greatness cannot be planned" by Kenneth Stanley - https://youtu.be/lhYGXYeMq_E?t=416

I see your point when it comes to paper publication in general, but I feel that your post is very unwarranted with respect to the original post: DeepMind has recruited top-quality theoretical researchers from public institutions, it is not just experimental work which your average Joe could do with a few guesses. These researchers published a lot of theoretical papers before they were recruited, and they still publish a lot of them now that they are working at DeepMind, but they have more computational hardware to apply their ideas.

Here is just one uncurated example of a publication: - https://deepmind.com/research/publications/Taylor-Expansion-... - https://arxiv.org/pdf/2003.06259.pdf

> An average Joe could probably write influential ML papers by gluing RNN/GAN layers to existing models and fiddling with the parameters until they beat current state of the art.

Right, but you have to remember there are legions of grad students doing exactly this so it ends up being quite competitive to churn out papers this way.

> The people doing math above algebra are few and the scene is dominated by "guess and check" style model tinkering.

"guess and check" is terribly ineffective with multi-day training runs. Brings us right back to the batch processing paradigm of the 1960s.

This feels much like the sentiment in the field about two years ago or so. While I feel like the "alchemy" storyline is still somewhat in play, most of the big important parts of the deep learning process have enough ideological linear approximators stacked around them that if you know what you're doing or looking at, you can jump to an unexplored trench with some reasonable feeling about whether you'll get something good or not. I feel like the "alchemy" approach is when people new to the field are innundated with information about it, and while I think that still holds, there very much is a well-understood science of principles in most parts of it.

There's the neural tangent kernel work that's achieved a lot, and the transformers themselves are really taking off a lot as the blockwise/lower rank approximation algorithms look more and more like circuits built off of basic, more well-established components.

"An average Joe could probably write influential ML papers by gluing RNN/GAN layers to existing models and fiddling with the parameters until they beat current state of the art. In fact, in NLP models, this is essentially what has happened with roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative but yet again built on existing models."

This feels like it trivializes a lot of the work and collapses some of the major advancements in training at scale down to a more one-dimensional outlook. Companies are doing both, but it's easy to throw money and compute at an absolutely guaranteed logarithmic improvement in results. It's not stupidity, it's just reducing variance in scaling known laws as we work on making things more efficient, which weirdly enough starts the iterative process of academics frantically trying to mine the expensive, inefficient compute tactics to flag plant their own materials.

With respect to you comment on protein folding and such, I feel you might have missed a lot of the major work in that aren more recently. There really and truly been some field-shattering work on that in combining deep learning systems with last-mile supervision and refinement systems. I'd posit that we're very much out of the wild west and in the mild, but still rambunctious west, if I were to put terms on it.

With reference to guess and check -- yes, that especially was prevalent and worked 2-3 years ago and I'd be in favor of advocating that it does still happen somewhat in a more refined fashion, but I personally believe we'd not get too far beyond the SOTA if we're not working (effectively) with your data manifold now and tightly incorporating whatever projections/constraints of that data distillation process into your network training procedure. I really do agree with you in that I think average Joe breakthroughs will happen and continue to benefit the the field, and I'd certainly agree that there's always going to be the mediocre paper churn of paper mills I think that you alluded to trying to justify their own existence as academics/paper writers, but I really do legitimately think there's enough precedent set in most parts of the field that you need to have some kind of thoughtful improvement to move forward (like AdaBelief, which is still terrible because they straight up lie about what they do in the abstract, even though the improvement of debiasing the variance estimates during training is an exceptionally good idea).

Just my 2c, hope this helps. I think we may have a similar end perspective from two different sides, like two explorers looking at the same peak from the different side of the mountain. :thumbsup:

Good write-up. Indeed I'm a novice tinkering with a decent gaming GPU :) . I was initially daunted by ML but the more I read I began to realize the field is quite accessible these days. Most of the time, you don't need to understand why or how this stuff works at a deep level. You just need a good feel of what might work and a training dataset.

Much of that is the enormous amount of work done plastering over complex GPU programming. But some of it is the tinkering nature of solving ML problems.

The field I'm most interested in right now for instance, NLP, is highly dataset dependent. It's fairly easy to exceed SoTA right now using open sourced models if you have a better, more specialized dataset than what's freely available.

Absolutely, couldn't agree more. If you want a secret, just find what scaling laws are there and find tunnels to bypass them. There's always a way to the secret garden, you just sometimes have to look long and hard... ;)

I started with my 1070 flat, and have had some people far, far, far smarter and more experienced than me help me understand a lot of the underlying mathematics a lot. Semi-supervised/bootstrapping may be a fun topic, if you can avoid the giant CAT trucks of the FAANG monoliths blazing through there, and there's always really good artisanal work to be done if you can prove certain mathematical conditions hold such that other (oftentimes counterintuitive and bizzare) operators still work, or work when they shouldn't before.

You could also get into the rat race of the *formers -- the Nyströmformer is quite spectacular and nearly linear, and yes, if you're hot on your feet and clever enough, you might be able to beat everything into submission.

Also, distrust every non-bayesian thing involving means and sigmas. Those are always ad hoc and beat the real data manifold into submission, which really does a disservice a lot of the time, I think. There's a lot to get around that (I suppose including the above, which I'd forgotten about, but there's always, uh, SeLU if you're looking for inspiration plus a phenomenal appendix. You want universal attractors? Set up and prove something that's more amenable to a good manifold structure than simply a certain distribution of activations -- that truly tells us nothing!)

Hope those are fun ideas -- and my deepest apologies if I was uncharitable to you in my former post. I went back and edited it for politeness but reading it again felt some of my earlier aggression fall through, and I'm certainly sorry about that -- I should be helping new folks, not being an aggressive gatekeeper against that.

In any case, so long as you're able to keep mathematical interest, there's always a nice hole to square yourself away into. Talk to a good accomplished research professional and they might be able to point you in fun directions (aside from my personal noobishness ;))

Let me know if any of those catch your eye and end up going anywhere, I'm happy to help when it moves the field forward! :)))

Minor variations on top of existing stuff and occasional leaps forward is most research though, it's not surprising ML research follows a similar pattern.

While I generally agree to some extent: roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative Researchers take inspirations from existing models of course and some BERT derivatives are trivial. However, XLnet is in it's own league, while the author (a genius chinese student) was inspired by BERT it is one of the few SOTA pré trained models to be not based on BERT and is actually an auto regressive one! Such difference allow it to be better at many things as it doesn't has to corrupt the tokens (from my shallow understanding). This model is two years old but is still sadly the one that ranks the most SOTA in key tasks e.g dependency parsing. And after all those time nobody cared enough to even test it on other foundational tasks (which is extremely sad and pathetic) like e.g coreference resolution. Sadly because of conformism effects almost zero researcher has created XLnet derivatives. Almost all researchers continue to search in the local minima that is BERT, which I find, immensely ironic.

While ad hoc empirical fine tuning is a big part of improving sota, mathematical genius can still enable revolutions e.g this recent alternative to classical backpropagation that is 300X faster with low accuracy loss https://paperswithcode.com/paper/zorb-a-derivative-free-back...

Not sure why you're being downvoted. I was about to swoop in and mention that the top level comment was wrong about XLNet being some bert based model but you beat me to it.

Sometimes HN is full of people who think they know what they're talking about but just don't. This is one of those times.

Honestly my comments that have negative Karma have generally much more useful truths than my comments that have positive Karma, this is almost systemic. This show how low quality the HN community is, epistemologically speaking. There are much less lazy communities out there like lesswrong.com but unfortunately they don't talk much about computer science

Interesting, but in not sure you're completely right about XLNET. I heard it takes an absurd amount of resources to train. Even more than the BERT variations. And this is likely why there's not a ton of interest in it

https://github.com/renatoviolin/xlnet XLnet running on very low end hardware (a single 8GB 2080 non ti) significantly outperform BERT large on e.g the reference question answering benchmarck: SQUAD 2 86% vs 81%

Nobody has even tried to create a spanXLnet (akin to spanBERT) How many years will be wasted before researchers get out of the BERT local minima? I'm a afraid it might last a decade