Agreed that this is another way of framing the problem of regularizing a network. Rather than starting with a big network and penalizing complexity, they are starting with a simple network and adding complexity. To that end, I'd've liked to see a comparison to dropout or L1/L2 regularization.