Why do you think that throwing compute at the problem is absurd? Arguably that's the approach that produced the resurgence of interest in CNNs.

Yes I think the whole point of Zoph and Le's research is to empirically demonstrate the desired trade off of compute cycles vs feature engineering

I'm not the original commenter, but I think it's fair to say that throwing compute at the problem is absurd from an academic point of view (where the goal is understanding), but not from a commercial point of view. Although there's still some academic value in throwing compute at a problem, just to see what's possible.

Can you link the paper of Zoph,Le that did the “1000 GPUs” thing? Thanks! I tried searching for it but apparently there was quite a few papers on parameter search.

Not GP, but you are probably looking for the "Neural Architecture Search" series [1] [2] [3]. First one uses something like 1k GPUs for a month, next one is a bit more reasonable, and the last one actually has a reasonable training time, and has comparable performance to DARTS (see ENAS in comparison tables).

1: https://arxiv.org/abs/1611.01578 2: https://arxiv.org/abs/1707.07012 3: https://arxiv.org/abs/1802.03268