News for hackers. Stuff that matters.
Re:Alphafold by nl on Thursday July 15, 2021 @08:51PM (#27851932)
> Most of the compute in ML research seems to be going into architecture search.

No it's not. Only Google spends significant time with automatic architecture search, and many people think this is really to try to sell cloud capacity.

> Once the architecture is found, training and net finetuning/transfer learning is comparatively cheap

Training isn't cheap for significant problems.

Getting the data is very expensive, and compute is a significant expense for large datasets.

> This implies we could see 10-100x gains in AI algorithms using today's hardware

Actually, most of the time we see 10-100% (percent! not times) gains from architecture improvements, whether they be manual or automatic.

But that is very significant, because a 10% improvement can suddenly make something useful that wasn't before.

Re:Alphafold by Tenoke on Thursday July 15, 2021 @11:28PM (#27852780)
>No it's not. Only Google spends significant time with automatic architecture search, and many people think this is really to try to sell cloud capacity.

Maybe not automatic architecture search but a lot does go into testing different architectures and changes to them in a more manual manner. Though yes, those tests are run on a smaller scale so for those huge models, training will be a bigger portion compared to architecture search than for smaller models.

Re:Alphafold by neolog on Thursday July 15, 2021 @11:05PM (#27852657)
parent's EfficientNet graph seems really dramatic. Is it misleading somehow?
Re:Alphafold by nl on Friday July 16, 2021 @08:21AM (#27855507)
It's not misleading assuming you know the field.

If you don't you might not realize they are comparing ResNet (designed for ultimate performance) vs EffcientNet to show how close it gets in accuracy in the FLOPs budget.

Note that the best accuracy for EffcientNet is roughly 1-2% better than ResNet/ResNeXT/SENet/etc but does have a much better FLOPs budget.

But these other architectures were never optimised for FLOPs. It wasn't event a consideration when designing them.

And EffcientNet is about a (manually designed) technique for scaling neural networks up in accuracy. Only EffcientNet-B0 is designed by AutoML, the others are scaled up. See the paper[1] for complete details.

Like-for-like should be against MobileNet etc. EfficentNet is better, but the comparison is more reasonable in general.

https://machinethink.net/blog/mobile-architectures/ is a really good overview.

[1] https://arxiv.org/pdf/1905.11946.pdf