It's very impressive as an AI demo. While it's far from perfect, it seems to have a much better grasp of English than previous examples I have seen. It's picking up on small subtleties, it's grammar and spelling are decent, and it's even remembering long strings of arbitrary characters. Which a difficult task for RNNs, which have to learn many "memory cells" from scratch using millions of floating point ops. And it's doing all that with an insect sized brain.

I don't know why this is. Perhaps training on dictionaries is a really good way to teach NN's about English. Who would have guessed?

the nature of the label bias present in rnn - the weights aren't learned by estimating the loss over the whole input/output, they are localized... representation helps a lot for the decisions but if you trained rnn-s differently - using the loss over the whole output - you would get better results