Very insightful comment, thank you. There's one other related thing I also find worthy of exploring further namely the population based training used by AutoML-Zero at the moment seems extremely simplistic, and there exist a lot of bleeding edge methods in that area which can tremendously improve outcomes of evolutionary algorithms, I've tweeted about them here (and at the AutoML-Zero people):

https://twitter.com/no_identd/status/1238565087675330560

And it doesn't seem unlikely that tweaking these would tremendously improve the outcomes. Combining that with what you've just described would… well, I'll leave that to the readers imagination. ;)