I don't know much about his origins, but surely that's not how he used his time when he was getting started, is it?

I'm guessing his work activities were far more conventional (though still well selected) and only after building wealth did his time allocation change.