News for hackers. Stuff that matters.
Re:How to train large deep learning models as a startup by jszymborski on Thursday October 07, 2021 @01:04PM (#28789148)
> "create a NN and stack more layers until you have your MVP"

I mean, that's a pretty good principled approach to a lot of ML problems.

Re:How to train large deep learning models as a startup by r-zip on Thursday October 07, 2021 @01:30PM (#28789542)
I think you have a different definition of "principled" from most people.
Re:How to train large deep learning models as a startup by jszymborski on Thursday October 07, 2021 @02:46PM (#28790529)
I'm very curious as to what part of that process is not explained by the principles by which we understand neural networks to work.

I invite the possibility I've gone this long misunderstanding the definition of "principled" in this context.

Re:How to train large deep learning models as a startup by r-zip on Friday October 08, 2021 @11:44AM (#28800515)
To me, taking "principled approach" means you understand and can justify the eventual outcome of the approach, or at least guarantee that the outcome satisfies some constraints. How would you justify the number of channels in each layer of a convolutional network? The number of self-attention heads in a transformer? The depth? Can you certify its prediction performance?

Yes, the "just add more layers" approach typically works (in a very narrow sense of the word "works"), but we don't really understand why. We likewise don't understand the failure modes of the system, and cannot engineer around them. Thus it's not really principled in my view.

Re:How to train large deep learning models as a startup by cardosof on Thursday October 07, 2021 @03:49PM (#28791212)
Only because currently ML is more alchemy than engineering. We mix stuff until we make gold while we can't explain why more parameters generalize better instead of overfitting.