News for hackers. Stuff that matters.
Re:The Unreasonable Effectiveness of Recurrent Neural Networks by digikata on Thursday May 21, 2015 @06:32PM (#9585772)
Human children have the great benefit of interactively learning from their parents and other humans raising them. Could we expect a child to learn to speak if they only heard recordings of existing speech with otherwise no human interaction/feedback - correcting them or offering customized and contextual new bits of information? It would be interesting to add feedback path for human corrective input. i.e. because it's direct interaction, feed it back but somehow weight it a little more than just another corpus input.
Re:The Unreasonable Effectiveness of Recurrent Neural Networks by tzs on Thursday May 21, 2015 @11:06PM (#9586428)
> Could we expect a child to learn to speak if they only heard recordings of existing speech with otherwise no human interaction/feedback - correcting them or offering customized and contextual new bits of information?

I once asked a similar question on some online forum [1] where many linguists hung out. My question was if an English-only speaking household left a general interest Spanish language TV station on most of the time when they weren't actively using the TV to watch something, so that their child received a very large exposure to Spanish language programming (news, sports, soap operas, sitcoms, movies, etc) from birth onward, would the child naturally learn Spanish?

I don't recall for sure what the linguists who responded said, but I think they all said the child would not learn Spanish from this.

[1] I have no recollection of where this was.

Re:The Unreasonable Effectiveness of Recurrent Neural Networks by spdionis on Friday May 22, 2015 @04:50AM (#9587179)
When I was little (5-7 years old) I had quite a few anime videos and magazines in italian sent to me by my parents who were abroad. Where I lived no one knew a single word of italian. I often watched and rewatched those videos and read those magazines without any other external input. I can tell you that doing that I easily learned the language. When I was 8 years old I also left for Italy and in two weeks of time I already started speaking fluently, albeit with a few mistakes.

If the child will actually watch the Spanish TV he will learn the language.

EDIT: Even now I often learn new japanese words (and remember them) just by watching animes. The difference is that now I have english subtitles but back then I had no subtitles, only the images to help me understand the meaning.

Re:The Unreasonable Effectiveness of Recurrent Neural Networks by digikata on Friday May 22, 2015 @11:39AM (#9588951)
But this is a little different, in the AI we want the ability to form syntactically correct sentences, but also some intelligence behind the sentences too. You as a human had another foundation of intelligence to lean on, your native language, and an understanding of the world outside of learning the Italian language. If you had no other human interaction, would you have learned any language? That's the more the situation of these AI algorithms.
Re:The Unreasonable Effectiveness of Recurrent Neural Networks by spdionis on Friday May 22, 2015 @12:36PM (#9589379)
I was only arguing that the child could actually learn Spanish, nothing more.

Not knowing basically anything about AI state of the art what stops us from feeding a RNN image data and text data and make it correlate them automatically by context? Just like a child learns words by hearing them many times in similar contexts so could a RNN.

I imagine the biggest problem is gathering and structuring the data. We humans receive lots of data and have lots of time to process it in our lives compared. And by lots I mean difference of a few orders of magnitude. It's amazing what this thing learns in just a few hours of processing.

Re:The Unreasonable Effectiveness of Recurrent Neural Networks by digikata on Friday May 22, 2015 @01:11PM (#9589662)
I agree the RNN performance is really amazing!
Re:The Unreasonable Effectiveness of Recurrent Neural Networks by geomark on Thursday May 21, 2015 @11:49PM (#9586501)
I've seen something like this in action with young kids who are given a tablet and stumble upon cartoons they like on YouTube, but in a different language. After they watch a few cartoons, YouTube's recommender system keeps offering them more cartoons in that language. And it isn't long before they start spouting words and phrases in that language.
Re:The Unreasonable Effectiveness of Recurrent Neural Networks by dzhiurgis on Friday May 22, 2015 @12:00AM (#9586521)
In Baltic states, majority of TV content either comes in native German or is US shows translated to Russian.

I've picked up quite a bit of Russian by watching Discovery channel this way.

Re:The Unreasonable Effectiveness of Recurrent Neural Networks by dwiel on Thursday May 21, 2015 @11:19PM (#9586449)
There are some recent examples where people have trained a collection of large nets which are then used to teach a smaller net. The smaller net can learn more quickly and finally achieves better performance than the large collection.

The methods involve providing more detailed feedback at each example. With most training data used now, we give a 0 or 1, does this example belong to this class. In the teacher networks, they were able to teach with more subtly: this is definitely not a car, it is very lizard like and a little snake like.

Re:The Unreasonable Effectiveness of Recurrent Neural Networks by kimbly on Tuesday May 26, 2015 @09:12PM (#9608705)
Do you have a reference?
Re:The Unreasonable Effectiveness of Recurrent Neural Networks by sinwave on Friday May 29, 2015 @01:56AM (#9622927)
Geoffrey Hinton gave results to this effect in a talk about "Dark Knowledge" [1]. Haven't seen any of these results published, though. I think he mentions something in the talk about NIPS rejecting the paper.

[1] - https://www.youtube.com/watch?v=EK61htlw8hY

Re:The Unreasonable Effectiveness of Recurrent Neural Networks by jpatokal on Friday May 22, 2015 @01:51AM (#9586711)
The TL;DR appears to be "no", interaction is necessary:

https://en.wikipedia.org/wiki/Language_acquisition#General_a...

Although for obvious reasons this is very hard to study experimentally:

https://en.wikipedia.org/wiki/Language_deprivation_experimen...

Re:The Unreasonable Effectiveness of Recurrent Neural Networks by eli_gottlieb on Friday May 22, 2015 @08:08PM (#9591492)
Proper language usage probably involves causal modelling, in which case intervention experiments are one of the only known ways to learn correctly.
Re:The Unreasonable Effectiveness of Recurrent Neural Networks by solve on Thursday May 21, 2015 @07:59PM (#9586033)
I'm convinced that voice pitch, syllable meter, hand gestures, facial gestures, and other forms of non-word emphasis are also crucial to figuring out basic phrase chunking and word types, with minimal interaction, early on.

Seems it would be far harder to infer the basic initial structure from just plain text.

Re:The Unreasonable Effectiveness of Recurrent Neural Networks by frozenport on Friday May 22, 2015 @12:19AM (#9586548)
Umm. When you train you train against a cost function...
Re:The Unreasonable Effectiveness of Recurrent Neural Networks by digikata on Friday May 22, 2015 @11:34AM (#9588930)
Well not exactly, if we had a cost function as intelligent as a human, the cost function to train the AI, would be an AI. Or maybe I'm completely off base here...