News for hackers. Stuff that matters.
Re:Why does deep and cheap learning work so well? by jdonaldson on Friday May 11, 2018 @08:43PM (#17051786)
Love to see more papers like this. I remember a recent paper that showed decent model performance when the model was allowed to pick its own activation functions. It was picking wacky things like sine waves. You'd like to learn something about the model by the way it configures itself, but right now we can only understand simple model features.
Re:Why does deep and cheap learning work so well? by ralusek on Friday May 11, 2018 @09:17PM (#17051934)
That's so weird, I can't help but feel like that would only ever result in it coming up with something that initially got it a better result and the rest of the network adjusted around it and is able to carry the weight of that weirdness.

What IS interesting is that it shows just how resilient the networks that come out of this really are that they can support such weirdo activations.

But there are other elements of introspection that should absolutely be handled by neural nets, things like choosing how deep a given network needs to be should be accomplished by other neural nets specifically looking for trends while training.

Re:Why does deep and cheap learning work so well? by jdonaldson on Saturday May 12, 2018 @06:24PM (#17056329)
Yeah, I agree about smarter parameter search and training. That was a big theme at ICLR this year.

Fwiw, here's the paper on RNN architecture generation referenced earlier.

https://arxiv.org/abs/1712.07316