News for hackers. Stuff that matters.
Re:Neural Networks, Types, and Functional Programming by colah3 on Thursday September 03, 2015 @11:55PM (#10168747)
TreeNets are also pretty popular. Word embeddings are also a form of weight tying, and can't always be described as part of an RNN. If you want, you can think of most attentional mechanisms as involving a form of weight tying, in addition to the RNN tying.

There are also more obscure things, like deep symmetry networks (where are really a close relative of conv nets, with convolution replaced by group convolution).

Re:Neural Networks, Types, and Functional Programming by cs702 on Friday September 04, 2015 @08:17AM (#10169905)
Thank you. It seems to me that every time a new state-of-the-art result in AI is announced, a deep composition of convolutional or recurrent "neural functional programs" is involved.

I don't see deep compositions of treenets or word embedding layers, which tend to be used instead stand-alone as simpler models or as preprocessing layers to deep networks. I'd have to think about attentional models.

This is not a criticism. Rather, it's my way of suggesting that we need more experimentation with more interesting compositions using a broader range of "neuron functional programs" -- which I believe is also one of your points.

And again, I think your essay is fantastic.

--

Edits: changed my wording to express what I actually meant to write.