As far as I know, a lot (most?) advances in the field are just trying new ideas that happen to work. Is this correct?
EDIT:
copying and pasting this from another answer:
I think its way different than that, those would just be precursors, and in cases like real analysis, superfluous. Instead it would look something like, I have a Universal Sentence Encoder architecture, but its not performing well on my data, aside from tweaking the training set, how can I take this architecture and change it to work better with my individual problem? The number of people on the planet that can do this successfully, without wasting months of time messing around with tensorflow is extremely small. But this is where the value is. These massive catch all models only work for the people creating them, just jamming them into any NLP model will always produce sub par and probably unusable results
I think humans can still invent new macro structures like CNN's...but humans are inherently shit at analyzing "what if we removed one neuron in the 2nd hidden layer?". The subtle tweaking is really best left to an automated recursion process.
Humans are better at seeing/inventing macro structures - such as adapting the unidirectional GPT to a bidirectional ELMO/BERT. After the invention, humans are generally pretty good at determining "whether" a network can be used to solve a particular task, although not infallible [1: Can BERT generate sentences from a prompt like GPT?]
But computers are once again often better at quickly determining whether which (ELMO, BERT, or GPT) perform better on a particular task for which they are all at least feasibly suited.
0: http://ai.googleblog.com/2019/08/efficientnet-edgetpu-creati...
1: https://ai.stackexchange.com/questions/9141/can-bert-be-used...