News for hackers. Stuff that matters.
Re:Google’s new voice recognition system works instantly and... by modeless on Tuesday March 12, 2019 @04:50PM (#19372870)
Re:Google’s new voice recognition system works instantly and... by gok on Tuesday March 12, 2019 @05:05PM (#19373021)
These are much much better links
Re:Google’s new voice recognition system works instantly and... by p1esk on Tuesday March 12, 2019 @05:26PM (#19373218)
Interesting that they're using RNN transducer. I thought everyone's moved to CNN lately.
Re:Google’s new voice recognition system works instantly and... by lawrenceyan on Tuesday March 12, 2019 @05:54PM (#19373447)
CNN's are only mentioned because of potential processing considerations as computationally, they are easier to deal with. But given the nature of speech recognition, which is so highly temporally correlated, it shouldn't be a surprise that a recurrent neural network would be used. This is pretty much exactly the purpose the RNN type of model architecture was designed for.

Also if you haven't looked into the properties of how exactly a RNN Transducer functions, I highly recommend doing so. They help resolve a great deal of problems that traditional RNNs and CNNs are unable to deal with.

Re:Google’s new voice recognition system works instantly and... by p1esk on Tuesday March 12, 2019 @06:13PM (#19373604)
I take it you haven't been following the field lately, have you? It is surprising because convnets just work better for speech recognition ([1] is the latest SOA). I'm guessing gated convolutional LMs are slower than RNN transducers when deployed on mobile. Can someone confirm?

[1] https://arxiv.org/abs/1812.06864

Re:Google’s new voice recognition system works instantly and... by GistNoesis on Tuesday March 12, 2019 @06:53PM (#19373914)
There is also the transformer approach (eventually with local attention to bound latency), (like I'm doing in my project (Work in Progress) : https://github.com/GistNoesis/Wisteria/blob/master/SpeechToT... ), though it's in the same line of thought as the convolutional CTC.

The RNN-T is a nice idea though, if I understand it correctly it's another approach to the alignment problem. In CTC, you are generating sequence like TTTTHHHEE CCCAAATT, which mean that your language model must deal with these repetitions, and you can't train using text without repetitions. In RNN-T you are learning to advance the cursor on either audio sequence or text sequence so as to maintain alignment, kind of like you do when you merge-sort two sorted lists, therefore it outputs THE CAT, and you can use a standard language model.

Re:Google’s new voice recognition system works instantly and... by JulianSlzr on Tuesday March 12, 2019 @10:53PM (#19375265)
We explored self-attention + CTC in our ICASSP 2019 paper (https://arxiv.org/abs/1901.10055). Our implementation uses internal infra, so we're a ways from releasing code :(

Hoping paper details suffice and help with the parameter search, and happy to respond to questions over e-mail. Would love to see an open-source implementation with local or directed attention built out!

Re:Google’s new voice recognition system works instantly and... by p1esk on Wednesday March 13, 2019 @03:05AM (#19376379)
Very interesting! Perhaps I should revise my original comment to “everyone is moving to transformers lately” :)
Re:Google’s new voice recognition system works instantly and... by p1esk on Tuesday March 12, 2019 @07:38PM (#19374192)
What is "convolutional CTC"?

Gated convolutions as LM is similar to RNN-T idea [1], but you have to deal with softmax, so I'm not sure how well this would work in practice, especially on a mobile processor.

[1] https://arxiv.org/abs/1612.08083

Re:Google’s new voice recognition system works instantly and... by blcArmadillo on Tuesday March 12, 2019 @09:35PM (#19374918)
I'm guessing Connectionist Temporal Classification: https://www.cs.toronto.edu/~graves/icml_2006.pdf
Re:Google’s new voice recognition system works instantly and... by vsef on Tuesday March 12, 2019 @06:36PM (#19373806)
The paper you site gets equal to SOA performance on Wall Street Journal and LibriSpeech test sets, both of which are clean, read speech, and not at all representative of what a phone or assistant recognizer deals with. The convnet described there also does not perform streaming recognition.

The primary reason to be interested in convnets for speech is computational parallelism, not because they have especially strong results for accuracy.

Re:Google’s new voice recognition system works instantly and... by p1esk on Tuesday March 12, 2019 @07:32PM (#19374149)
I'm curious why people think that recurrent architectures are somehow more noise-tolerant. Where did this come from?
Re:Google’s new voice recognition system works instantly and... by vsef on Tuesday March 12, 2019 @07:51PM (#19374259)
They're not that I know of. But the paper cited is just showing relative parity or slight improvement on relatively toy examples. The claim that convnets are the clear winner for speech in general/what everyone is doing now is just not true.

I work in the field, a more accurate summary would be that there are a number of viable architectures that currently get fairly similar accuracy, but that have other pros/cons with respect to streaming, memory use, parallelism, model size, integration with external language models and context, complexity of the decoder, friendliness to different types of hardware etc.

Re:Google’s new voice recognition system works instantly and... by p1esk on Tuesday March 12, 2019 @08:05PM (#19374363)
Ok, so what are the advantages of RNN over CNN based models for speech to text, with respect to any one of those factors you mentioned?
Re:Google’s new voice recognition system works instantly and... by vsef on Tuesday March 12, 2019 @10:08PM (#19375074)
Well for example, some comparisons to the CNN paper you pointed to:

- No comparison is given of number of model parameters. If optimizing strictly for model size, RNNs tend to be nice and compact.

- The computational advantage of the CNN at training time is throughput. The advantage of RNN at decoding time is streaming latency. Running the CNN frame by frame as they are received removes the ability to run frames in parallel and if the CNN is larger, it will run slower, and depending on its receptive fields it may not even stream well at all.

- That particular CNN system uses a strictly external LM that is not jointly trained and has an additional hyper parameter at decoding time to weight the LM that requires additional tuning.

- It is still autoregressive in the beam search, so the LM will still be run many times sequentially adding tokens just like an RNN LM, and is likely to be more expensive. The throughput advantage a conv lm has in scoring whole sentences is totally lost. In fact, there doesn't seem to be anything special about the choice of a conv lm for that paper except that it is fun to make all the parts convolutional.

- CNNs frequently require more total flops, but are high throughput on eg a GPU because they expose so much parallelism. On an embedded CPU this can be a bad tradeoff.

As a side note, there's no reason that CNN architecture, which in the paper is trained with a close relative of CTC and is decoded identically to a RNN CTC AM + external LM, couldn't be trained as an RNN transducer. Despite the name neither the am nor lm have to actually be RNNs.

Re:Google’s new voice recognition system works instantly and... by sdenton4 on Tuesday March 12, 2019 @11:38PM (#19375484)
Haha, I ran a straight convolution net as an encoder for asr in a toy project while learning seq2seq. Worked fine in the small datasets I was working with, like the voice commands set...
Re:Google’s new voice recognition system works instantly and... by p1esk on Wednesday March 13, 2019 @02:51AM (#19376318)
Thank you for the detailed answer. This is exactly what I was looking for when starting this thread.
Re:Google’s new voice recognition system works instantly and... by pakl on Tuesday March 12, 2019 @05:47PM (#19373393)
Feedforward CNNs cannot tolerate as much noise and real-world variability as RNNs.
Re:Google’s new voice recognition system works instantly and... by p1esk on Tuesday March 12, 2019 @05:59PM (#19373488)
Citation?
Re:Google’s new voice recognition system works instantly and... by robbomacrae on Tuesday March 12, 2019 @08:28PM (#19374510)
There is this new type of trolling where someone just puts in bare minimal counterarguments and asks for citations knowing full well it's labour intensive to do so. If your not trolling.. your not doing a good job of avoiding these comparisons. The onus should be on you here to provide citations first that dissprove the person your replying to.
Re:Google’s new voice recognition system works instantly and... by p1esk on Tuesday March 12, 2019 @08:46PM (#19374642)
I'm not trolling. The statement that RNN are somehow more noise-tolerant than CNNs does not make sense to me, and is not based on any literature about noise tolerance in NNs that I'm familiar with. Also, no arguments have been provided as to why this could possibly be the case.
Re:Google’s new voice recognition system works instantly and... by pakl on Tuesday March 12, 2019 @09:47PM (#19374982)
Hi p1esk, RNNs can be more tolerant to noise because they can learn transient or dynamic attractors. If the inputs move an RNN into an attractor, small changes due to noise make little difference to the state.

Recurrence can help with robustness in some other very important ways as well.

Citations for this dates from the 80s and 90s. I don't know the best reference offhand. You could look at some old Hinton stuff if you're a fan. Lots published on this.

Re:Google’s new voice recognition system works instantly and... by p1esk on Wednesday March 13, 2019 @03:32AM (#19376502)
That’s handwaving. The only way to find out if rnns are more robust to noise than cnns is to test them on the same task, with the same inputs and the same noise. Preferably using similar number of parameters and achieving similar accuracy before noise is applied. Then gradually increase the amount of noise, and compare the impact.

Nothing like this has been published AFAIK.

After you have the results of this experiment you can try to explain them with attractors and what not, but I would be surprised if there was much difference. Would make a good paper though!

Re:Google’s new voice recognition system works instantly and... by rocky1138 on Tuesday March 12, 2019 @09:36PM (#19374925)
> The onus should be on you here to provide citations first that dissprove the person your replying to.

Unless you're dealing with opinions, I disagree. The onus is on the person trying to give evidence to actually give evidence.

Re:Google’s new voice recognition system works instantly and... by taneq on Wednesday March 13, 2019 @12:31AM (#19375747)
There was a post here 9 days ago discussing "intellectual DoS attacks" which basically work like this: https://news.ycombinator.com/item?id=19293036
Re:Google’s new voice recognition system works instantly and... by fwip on Tuesday March 12, 2019 @11:15PM (#19375388)
Closely related to sea-lioning, in my opinion.
Re:Google’s new voice recognition system works instantly and... by stunami on Wednesday March 13, 2019 @05:59AM (#19377203)
Absolutely.
Re:Google’s new voice recognition system works instantly and... by pault on Tuesday March 12, 2019 @09:08PM (#19374768)
Sorry to go on a tangent, but this is the first time I've heard the word "transducer" outside of a conversation about clojure. Is it the same concept?
Re:Google’s new voice recognition system works instantly and... by crdrost on Tuesday March 12, 2019 @11:45PM (#19375531)
Not quite. A Clojure transducer from A to B is pure function which takes As to lists of Bs. To apply it to a sequence of As you flatMap it, so you apply it to each A in sequence and get a sequence of Bs which you splice into your final resulting sequence. Maps and filters are special cases, for example

    const mapTrans = fn => function* (x) {
      yield fn(x);
    };
    const filterTrans = predicate => function* (x) {
      if (predicate(x)) {
        yield x;
      }
    };
    const dupeTrans = n => function* (x) {
      for (let i = 0; i < n; i++) {
        yield x;
      }
    };
Clojure just observed that an isomorphism of these functions under Church encoding looked like f<B> -> f<A> for a special parametric type f, so could be composed with ordinary function composition, albeit backwards.

An RNN transducer is fundamentally three functions:

One takes a list of recent As to some C1.

Another takes a list of recent Bs to some C2.

A final one takes a C1 and C2 to a B.

Rather than mapping over each input independently the RNN-T is allowed to learn something about the relationship of recent outputs to the next output, and the relationship of nearby inputs. Clojure transducers thus have an order insensitivity that RNN transducers are allowed to be sensitive to.

Re:Google’s new voice recognition system works instantly and... by braindead_in on Wednesday March 13, 2019 @12:45AM (#19375804)
DeepSpeech 3 from Baidu also uses transducers. http://research.baidu.com/Blog/index-view?id=90.