News for hackers. Stuff that matters.
Re:Evaluating the Search Phase of Neural Architecture Search by circuithunter on Thursday November 21, 2019 @09:01PM (#21601808)
This is a really nice paper which asks some critical questions for the future of NAS research.

However, it's important to note that this paper doesn't show that NAS algorithms as originally designed, with completely independent training of each proposed architecture, are equivalent to random search. Rather, it shows that weight sharing, a technique introduced by ENAS [1] which tries to minimize necessary compute by training multiple models simultaneously with shared weights, doesn't outperform random baselines. Intuitively, this makes sense: weight sharing dramatically reduces the number of independent evaluations, and thereby leads to far less signal for the controller, which proposes architectures.

The paper itself makes this fairly clear, but I think it's easy to misinterpret this distinction from the abstract.

[1] ] H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean. Efficient neural architecture search via parameter sharing. ICML, 2018.

Re:Evaluating the Search Phase of Neural Architecture Search by hmwhy on Friday November 22, 2019 @12:16AM (#21602888)
Thank you for pointing that out! I did misinterpret that part of the abstract!

On a perhaps related note, it seem a bit surprising to me because when I first started with neural networks about a year ago, I tried to shortcut hyperparameter search by reusing weights and noticed that independently trained models with the same hyperparameters would produce model with different performance. I naively assumed that such correlation is something I don't want and it was something that everyone knows about so I just moved on.

Edit: typo (pointed --> pointing)

Re:Evaluating the Search Phase of Neural Architecture Search by heyitsguay on Friday November 22, 2019 @12:25AM (#21602947)
I've noticed this too. I've got a paper coming up on the arxiv soon that discusses this phenomenon, and structured random architecture search, in the context of semantic segmentation networks.
Re:Evaluating the Search Phase of Neural Architecture Search by bjornsing on Friday November 22, 2019 @02:59AM (#21603717)
> On average, the random policy outperforms state-of-the-art NAS algorithms.

Ouch... I guess random policy is the state-of-the-art then, no?

Re:Evaluating the Search Phase of Neural Architecture Search by corporateslave5 on Thursday November 21, 2019 @11:09PM (#21602330)
The problem with neural architecture search is that we don’t know what to optimize on. We need some kind of topological understanding of the data to help build algorithms that can optimize performance. Clearly passing in error rates is not doing the trick
Re:Evaluating the Search Phase of Neural Architecture Search by machinelearning on Friday November 22, 2019 @12:57AM (#21603164)
I agree, I've been thinking about this too. Do you want to work together on this?
Re:Evaluating the Search Phase of Neural Architecture Search by dchichkov on Friday November 22, 2019 @02:16AM (#21603524)
Should this be a cause to correct affected papers? If not corrected, maybe even retract?

I understand that there is still clear value in these papers (defining search space, achieving state of the art).

But it does seem, there is some partially broken science there (not evaluating against baselines, algorithm that performs same as a random baseline).

Re:Evaluating the Search Phase of Neural Architecture Search by Roark66 on Friday November 22, 2019 @06:45AM (#21604729)
I wish they wrote more about how the search space is created. There is appendix A that talks a little bit about search space representation, but doesn't give any details about how the search space is created in the first place. Is it choosing the architecture from a set of given manually created architectures or is it actually creating them at random?

There are some references to other papers so perhaps the details are those papers.

Re:Evaluating the Search Phase of Neural Architecture Search by sbierwagen on Friday November 22, 2019 @04:58PM (#21610557)
Yo dog, we heard you like neural architecture search, so we wrote a NAS for your NAS so you can optimize your hyperhyperparameters while you optimize your hyperparameters.