Randomly wired networks reminds me of a classic AI koan:

> In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.

> “What are you doing?”, asked Minsky.

> “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied.

> “Why is the net wired randomly?”, asked Minsky.

> “I do not want it to have any preconceptions of how to play”, Sussman said.

> Minsky then shut his eyes.

> “Why do you close your eyes?”, Sussman asked his teacher.

> “So that the room will be empty.”

> At that moment, Sussman was enlightened.

I don't see how that relates. You don't stick to a single random wiring, you only make the wiring random so that the space of possible wirings of the network is not constrained by the classical sequential paradigm.

Except random search was found to outperform state-of-the-art neural architecture search algorithms:

https://arxiv.org/abs/1902.08142

The story is more about how a random set of weights WILL have preconceptions of how to play. If you look at condition numbers or spectra of random normal matrices they are very much not random.

Back in engineering school, a million years ago, my partner and I used simulated annealing to “design” a digital circuit implementation in CMOS (a 32 bit CRC), minimizing various parameters like wire length to optimize its function. It worked shockingly well when simulated.

I am a huge fan of using randomized starting states and then allowing the computer to discover the best architecture. It produces, if nothing else, surprising results.