I do have a question, but first I must say that as I was reading your post I found myself nodding in agreement again and again and again. Yes, the Silicon-Valley-centric AI research community (including UK-based DeepMind and its surrounding satellites) tends to overlook or even ignore important research from elsewhere. Yes, "algorithmic creativity" and focused tinkering often prove more important than computational resources. Yes, innovation in deep learning does not require big data nor massive hardware, but "engineers are drawn to using the biggest datasets they can get, on the biggest machines they can access, like moths flitting around a bright light" (so true!). And yes, "genuine advances [have] consistently come from doings things differently, not doing things bigger." Ah-men!

With that out of the way, here's my question:

Have you guys tried or managed to achieve learning super-convergence with attention or residual attention models?

No I haven't tried yet. One of our students has started experimenting with language models (AWD LSTM) and getting encouraging results (about 3x speed improvement), but not with attention.

I really hope that these results will help encourage more people to both see where and how super-convergence can be achieved. I suspect that we're only scratching the surface of what's possible.

Thank you for the prompt answer -- and for your intellectual generosity.

I too hope these results encourage more people to see where/how super convergence can be achieved.

FWIW, trying to achieve it with fully attentional models is on my "R&D things to try" list at work.

Hi Jermey... Great article. I have a labeled product image dataset (thumbnails, and various sizes) that I'd like to use for the model training. Can you recommend any lightweight methods for commodity hardware?

You can use the approach we used to win the CIFAR 10 training cost section of the competition. If you use fastai/pytorch, then it's ~5 lines of code. Check out lesson 1 of http://course.fast.ai for the basic approach, but when calling `fit()`, add the param `use_clr_beta=(20,20,0.95,0.85)` which will enable 1cycle, and should allow of super convergence. Then train with SGD with a really high learning rate (somewhere from 1-3, generally).

Excellent. I will check it out.

How important was momentum cycle (The decrease then increase)?

Unrelated but it would be great if you could answer: In general, how important is momentum tuning and what are the heuristics for the same?

Based on Smith's paper and our experiments, it seems pretty important. But there's still much research to be done to understand momentum tuning and related concepts.

How important was momentum cycle (The decrease then increase)?

Unrelated but it would be great if you could answer: In general, how important is momentum tuning and what are the heuristics for the same?

Does the solution use pre trained Resnet models?

No you can't use pre-trained models in this competition.

Thanks for the prompt response. Do you use data augmentation? Sorry for all the questions which could maybe be answered by looking at the code, but I am just getting started with pytorch etc and I am trying to understand the exact input format.

Yes data augmentation is critical

Is the code used for the competition available?

Yes it's linked from the competition web site. It'll be integrated into the fastai library in the coming weeks.