Ask HN: Are modular neural networks an interesting avenue for further research?
In principle, the hierarchical organization could allow us to make sense of more complex problem spaces and reach a higher functionality, but it seems difficult to find examples of concrete research done in the past regarding this. I've found a few sources:
https://www.teco.edu/~albrecht/neuro/html/node32.html
https://vtechworks.lib.vt.edu/bitstream/handle/10919/27998/etd.pdf?sequence=1&isAllowed=y
A few concrete questions I have:
Are there any tasks where MNNs have shown better performance than large single nets?
Could MNNs be used for multimodal classification, i.e. train each net on a fundamentally different type of data, (text vs image) and feed forward to a higher level intermediary that operates on all the outputs?
From a software engineering perspective, aren't these more fault tolerant and easily isolatable on a distributed system?
Has there been any work into dynamically adapting the topologies of subnetworks using a process like Neural Architecture Search?
Generally, are MNNs practical in any way?
Apologies if these questions seem naive, I've just come into ML and more broadly CS from a biology/neuroscience background and am captivated by the potential interplay.
I really appreciate you taking the time and lending your insight!
Automatically Composing Representation Transformations as a Means for Generalization https://arxiv.org/abs/1807.04640
Modular meta-learning https://arxiv.org/abs/1806.10166
Omega: An Architecture for AI Unification https://arxiv.org/abs/1805.12069
Cortex Neural Network: learning with Neural Network groups https://arxiv.org/abs/1804.03313
Evolutionary Architecture Search For Deep Multitask Networks https://arxiv.org/abs/1803.03745
A semi-recent success story is Tacotron 2, a speech synthesizer that has one model to convert text to a spectrogram, and a second model to convert that spectrogram into pcm (waveform) data.
Also, Francois Chollet, the author of the popular deep learning library Keras, has some interesting thoughts on the future of ML along these lines that I find compelling: https://blog.keras.io/the-future-of-deep-learning.html
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer https://arxiv.org/abs/1701.06538
You break big tasks into smaller tasks, which are generally manageable by people with 4-year degrees. An AI expert may manage or guide the larger project, but the sub-tasks shouldn't normally require an AI expert.