I think that's valid, however, I think the modular approach would allow for a lot more cross domain flexibility because I suspect that a large subset of disparate tasks do actually decompose themselves into discrete subtasks that can be useful across domains.

Adding to that, if you can compose multiple sub networks together then you've really got something. You can build a lot of different buildings from bricks without needing to invent new kinds of brick basically.

For instance, think about the large number of domains that robust computer vision would be useful in. Then think about the fact that if the computer understands the 3D space around it, it can hand that model off to a network that does predictive physics simulation. Now you've got something that would be useful across a extremely wide range of domains.