DeepMind achieves SOTA image recognition with 8.7x faster training

Re:DeepMind achieves SOTA image recognition with 8.7x faster... by throwaway189262 on Sunday February 14, 2021 @09:11PM (#26138287)

Good write-up. Indeed I'm a novice tinkering with a decent gaming GPU :) . I was initially daunted by ML but the more I read I began to realize the field is quite accessible these days. Most of the time, you don't need to understand why or how this stuff works at a deep level. You just need a good feel of what might work and a training dataset.

Much of that is the enormous amount of work done plastering over complex GPU programming. But some of it is the tinkering nature of solving ML problems.

The field I'm most interested in right now for instance, NLP, is highly dataset dependent. It's fairly easy to exceed SoTA right now using open sourced models if you have a better, more specialized dataset than what's freely available.

[ Parent | on HN ]

Re:DeepMind achieves SOTA image recognition with 8.7x faster... by tbalsam on Monday February 15, 2021 @03:31AM (#26140443)

Absolutely, couldn't agree more. If you want a secret, just find what scaling laws are there and find tunnels to bypass them. There's always a way to the secret garden, you just sometimes have to look long and hard... ;)

I started with my 1070 flat, and have had some people far, far, far smarter and more experienced than me help me understand a lot of the underlying mathematics a lot. Semi-supervised/bootstrapping may be a fun topic, if you can avoid the giant CAT trucks of the FAANG monoliths blazing through there, and there's always really good artisanal work to be done if you can prove certain mathematical conditions hold such that other (oftentimes counterintuitive and bizzare) operators still work, or work when they shouldn't before.

You could also get into the rat race of the *formers -- the Nyströmformer is quite spectacular and nearly linear, and yes, if you're hot on your feet and clever enough, you might be able to beat everything into submission.

Also, distrust every non-bayesian thing involving means and sigmas. Those are always ad hoc and beat the real data manifold into submission, which really does a disservice a lot of the time, I think. There's a lot to get around that (I suppose including the above, which I'd forgotten about, but there's always, uh, SeLU if you're looking for inspiration plus a phenomenal appendix. You want universal attractors? Set up and prove something that's more amenable to a good manifold structure than simply a certain distribution of activations -- that truly tells us nothing!)

Hope those are fun ideas -- and my deepest apologies if I was uncharitable to you in my former post. I went back and edited it for politeness but reading it again felt some of my earlier aggression fall through, and I'm certainly sorry about that -- I should be helping new folks, not being an aggressive gatekeeper against that.

In any case, so long as you're able to keep mathematical interest, there's always a nice hole to square yourself away into. Talk to a good accomplished research professional and they might be able to point you in fun directions (aside from my personal noobishness ;))

Let me know if any of those catch your eye and end up going anywhere, I'm happy to help when it moves the field forward! :)))

[ Parent | on HN ]