Re:Google’s new voice recognition system works instantly and...
by JulianSlzr on Tuesday March 12, 2019 @10:53PM
(#19375265)
We explored self-attention + CTC in our ICASSP 2019 paper (https://arxiv.org/abs/1901.10055). Our implementation uses internal infra, so we're a ways from releasing code :(
Hoping paper details suffice and help with the parameter search, and happy to respond to questions over e-mail. Would love to see an open-source implementation with local or directed attention built out!
Re:Google’s new voice recognition system works instantly and...
by p1esk on Wednesday March 13, 2019 @03:05AM
(#19376379)
Very interesting! Perhaps I should revise my original comment to “everyone is moving to transformers lately” :)
Re:Google’s new voice recognition system works instantly and...
by p1esk on Tuesday March 12, 2019 @07:38PM
(#19374192)
What is "convolutional CTC"?
Gated convolutions as LM is similar to RNN-T idea [1], but you have to deal with softmax, so I'm not sure how well this would work in practice, especially on a mobile processor.
Re:Google’s new voice recognition system works instantly and...
by blcArmadillo on Tuesday March 12, 2019 @09:35PM
(#19374918)
I'm guessing Connectionist Temporal Classification: https://www.cs.toronto.edu/~graves/icml_2006.pdf
The RNN-T is a nice idea though, if I understand it correctly it's another approach to the alignment problem. In CTC, you are generating sequence like TTTTHHHEE CCCAAATT, which mean that your language model must deal with these repetitions, and you can't train using text without repetitions. In RNN-T you are learning to advance the cursor on either audio sequence or text sequence so as to maintain alignment, kind of like you do when you merge-sort two sorted lists, therefore it outputs THE CAT, and you can use a standard language model.