News for hackers. Stuff that matters.
Re:Why Neurons Have Thousands of Synapses: a Theory of Seque... by mjpuser on Thursday November 05, 2015 @04:13AM (#10512381)
Hawkins' mantra is sequence based memory. I love this guy and his work. And although I think his company has had success with sequence based AI, it still doesn't totally answer how the brain works. His model basically works by taking encoded symbols and being able to predict the next encoded symbol. This works great for things with patterns (power consumption, traffic patterns, etc), but it doesn't work with language. You will not be able to predict what I am going to say, or predict my answer based on the question, at least not very well. The problem is that he needs to figure out meta computation. His product, nupic, would take in a string of words, which gets encoded into a semantic representation, and tries to predict what's next. But concepts are larger than a single word, and until it can build a working semantic representation of what it's currently processing, it will not prove to be a useful model in describing how the brain works, or at least being able to compare it to our brain.
Re:Why Neurons Have Thousands of Synapses: a Theory of Seque... by RobertoG on Thursday November 05, 2015 @06:43AM (#10512795)
If you study the Numenta algorithm, you will see that the prediction is not only for the next steep and that the prediction, is forward feed to another layer where it becomes the input of a more general and stable prediction.

Also, there is strong evidence that prediction is the main (if not the only) function of the brain. I would recommend Hohwy's book (https://books.google.fr/books/about/The_Predictive_Mind.html...) for an explanation of this idea.

Re:Why Neurons Have Thousands of Synapses: a Theory of Seque... by mietek on Thursday November 05, 2015 @06:24AM (#10512752)
> This works great for things with patterns, but it doesn't work with language.

Sure it does.

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

> Hawkins' mantra is sequence based memory.

Note that Hawkins wrote “On intelligence” in 2005.

Schmidhuber wrote “Learning complex extended sequences using the principle of history compression” in 1992, and “Long short-term memory”, with Hochreiter, in 1997.

Today, a LSTM-based RNN can answer your Gmail.

http://people.idsia.ch/~juergen/rnn.html

http://googleresearch.blogspot.com/2015/11/computer-respond-...

Re:Why Neurons Have Thousands of Synapses: a Theory of Seque... by mjpuser on Thursday November 05, 2015 @09:30AM (#10513458)
Sorry! To clarify that it doesn't work with language, I mean you can't ask it a question and then expect it to come up with an answer. For most meaningful questions, you can't sequence an answer from it's question in a general way. This might be in part because Hawkins isn't trying to mimic the brain, but learn about how it works to apply the algorithms it uses to our needs/wants.
Re:Why Neurons Have Thousands of Synapses: a Theory of Seque... by p1esk on Thursday November 05, 2015 @03:59PM (#10516079)
you can't ask it a question and then expect it to come up with an answer

Why not? If the model was trained on a large enough corpus of texts, it will have seen lots of answers to your question, or similar questions. It can, in principle, extract the important features of those answers, and present them to you as an answer. This is pretty much how the majority of humans would answer "the most meaningful" questions.

Re:Why Neurons Have Thousands of Synapses: a Theory of Seque... by mjpuser on Thursday November 05, 2015 @04:49PM (#10516350)
Nupic showcases it's "What did the fox eat" use case which uses cortical.io's SDRs of words. In this example it "asks the question" what does the fox eat after teach it what other animals eat, and without ever seeing fox before, it's able to accurately say what it eats since it groups foxes with coyotes, etc, which have a similar SDR. However, this is a sequence... They fed it sentences only in the form "a eats b" and then fed it "fox eats", and it replied "rodent" or whatever... The SDR's have to be sequences in order for the HTM to work. So if you ask "Guess how many balls are in this jug", you have to do an implied computation to guess how many possible balls fit inside the jug, and that implied computation is not the word sequence (or sdr sequence). Even if you gave it an infinite amount of similar questions and answers like this, it would never be able to figure out a general way to answer the question, which means you could always stump it by giving a different sized jug + different sized ball.
Re:Why Neurons Have Thousands of Synapses: a Theory of Seque... by p1esk on Thursday November 05, 2015 @05:19PM (#10516533)
H in HTM stands for "Hierarchical". This means, that it can, in principle, construct more general, more abstract SDRs (patterns) based on many low level SDRs it sees. Thus it's plausible to see forming of ideas out of sentences, or something similar. This is how it can build a world model, and then run your input through multiple levels of abstraction, as many as needed to give a high confidence answer.
Re:Why Neurons Have Thousands of Synapses: a Theory of Seque... by mjpuser on Friday November 06, 2015 @01:59PM (#10521249)
My example was also spoon feeding questions and answers to the HTM. If you were to just give it a corpus of text, and then ask it a question about that text, it would not be able to formulate an answer that has any meaning, and follow proper English grammar. If you disagree, you can respond with a working example.
Re:Why Neurons Have Thousands of Synapses: a Theory of Seque... by p1esk on Friday November 06, 2015 @11:16PM (#10523684)
If you disagree, you can respond with a working example.

I am the working example. I can read texts and answer questions. HTM is an attempt to explain how I do that. Obviously, it's an incomplete theory, but its main ideas make sense to me. You got a better theory?

Re:Why Neurons Have Thousands of Synapses: a Theory of Seque... by mjpuser on Monday November 09, 2015 @12:58PM (#10534572)
I don't have a better theory, but that wasn't my point. It may be that there is a setup to read and understand language, but there has never been code presented based on this technology that does it. This is why I'm interested to see how you could set it up to make it work. If you could prove me wrong, then great! I'd love to see it.
Re:Why Neurons Have Thousands of Synapses: a Theory of Seque... by p1esk on Monday November 09, 2015 @10:20PM (#10537421)
HTM is a theory. It's very incomplete at this point, because our understanding of the brain is very incomplete. Numenta published their code as they discover more and more about the brain. Once the system is complete enough to process language, the code will be able to process language.
Re:Why Neurons Have Thousands of Synapses: a Theory of Seque... by colhom on Thursday November 05, 2015 @04:27PM (#10516217)
A noamrl peosrn can uranesntdd this whuiott too much dftfliciuy . I scseput there is hveay rniaelce on the ecixtetapon of waht wdors lileky come nxet . I thnik taht liniestng to seceph wkors the same wya.
Re:Why Neurons Have Thousands of Synapses: a Theory of Seque... by versteegen on Thursday November 05, 2015 @08:39AM (#10513192)
Sure, Numenta seemed heavily focused on temporal sequences, but in "On Intelligence" Hawkins describes a more general picture. Prediction should be done not only on the past but on the context. The context includes other variables that also have to be predicted (e.g. predict the next word in a sentence based on a prediction of where this conversation is going)

I think that predicting the next word in a sentence does actually get to the point of the issue, which is why the Hutter Prize [0] is defined the way it is.

The human brain does: (1) operate sequentially on words (2) keeps an internal state that contains semantic information about what's been heard/seen (3) learns the semantic representation (4) is excellent at predicting the next word in a sentence, and continually does so (5) it's difficult to argue it's not learning to predict the next word in a sentence (ie optimised to that task).

Looking at that list, I'd say the only differences between what the brain is doing and what algorithms like these (in general) are doing are just the particular algorithms for learning the semantic representations in working memory and performing prediction, and less importantly the form of the input (humans have more cues available, including taking actions to get more information)

(Admittedly parsing a complicated sentence can require jumping backwards, but spoken language will have a very limited nesting level, and this just means including a small piece of the input in the working memory.)

In the last week I've actually been reading about algorithms for this (though I didn't know NuPIC was applied to sentences, thanks very much for mentioning it). Aside from the range of RNN-based systems that mietek mentioned, here are a couple more recent papers, related to word2vec, on representing the semantic and syntactic content of a sentence as vectors: [1], [2]. Both are based on optimising for prediction of the next words in a sequence. Now I agree it's dubious to try to reduce a sentence to a small fixed length representation, whether that's e.g. 2400 real numbers in [2], or a pattern of neuron activations in a RNN, but from the experimental results (which admittedly aren't always so great) they seems to be encoding something meaningful, though far too much information gets thrown away.

[0] http://prize.hutter1.net

[1] Quoc V. Le, Tomas Mikolov, 2014, Distributed Representations of Sentences and Documents, http://arxiv.org/abs/1405.4053

[2] Ryan Kiros et al, 2015, Skip-Thought Vectors, http://arxiv.org/abs/1506.06726

Edit: added a bit more about context