avallet

I am not quite sure there is such a thing. :p I was playing a bit loose with the mathematics here, and trying to find some more intuitive way to explain "XY^T is full-rank", but it got confusing. Sorry about that. I will edit my initial post accordingly. (Ah, can't edit it seems, oh well)

Linear correlation. Actually, it seems to be a bit more general than just uncorrelated, i.e. if the input is a m by n matrix X and ground truth a k by n matrix Y, the author requires that XX^T and XY^T to be full-rank. A whitening transformation would yield the identity matrix for XX^T, but that's a bit stronger than what's strictly necessary. My interpretation of XY^T being full-rank meaning X and Y being uncorrelated might indeed be mistaken.

As I understand it, the main result of the paper relies on these 4 assumptions:

- That the dimensionality of the output of the network is smaller than that of the input. That is usually the case in image recognition, where the image is width x height x channels dimensional, while the output is usually a much smaller number of label-wise probabilities. It probably isn't the case when you generate data from some smaller representation, e.g. with autoencoders, image generation, etc.

- That the input data is decorrelated, and that the input data is uncorrelated with the output ground truth. The former can easily be obtained via a whitening transformation in many cases in practice. I am not quite sure about the latter.

- That whether a connection in the network is activated or not is random with the same probability of success across the network. Active means the ReLU activation function has output greater than 0. Many people initialize weights in the network with some 0-mean random variable and some constant bias, in which case that assumption should hold true at the beginning of training. Whether that assumption holds throughout training could easily be verified empirically - i.e. by monitoring the network's activation.

- That the network activations are independent of the input, the weights and each other. That's obviously not completely true - the network activations of a given layer are a function of the previous layer's activations and weights, and ultimately of the input in the first layer. With large enough networks, this may hold sufficiently in practice - any single activation should not depend very significantly on any other single variable.

I may have missed something in interpreting the maths, any comment is appreciated. From a practical standpoint, especially for computer vision, these assumptions seem quite reasonable. I am not qualified however to comment on the proof of this result, so I would wait on peer review. Still, it is heartening to see the theory of deep learning finally catching up with practice!

I've been living in Japan for a couple years, and this reflects my experience too with regards to most people I've met. However, it's not just the government, but also a powerful and ageing elite which shares these views. Admittedly not people you'll meet everyday, yet they wield obvious political influence.

Anecdotally, I hadn't met any such people until yesterday, where I had to attend a rotary international conference for a scholarship I receive from them. The audience was composed of around 700 people, most of them CEOs of local corporations, politicians or otherwise wealthy and connected people. The average age must have been somewhere in the late fifties/early sixties. The invited speaker was Masahiko Fujiwara [1], a mathematician turned essayist, who probably should have stuck to number theory. It was basically an hour and a half of chauvinist propaganda, hailing the past greatness of Japan, the superiority of the Japanese people, and mocking (I quote and translate) "Americans, Europeans and white people" at every turn with racial stereotypes. And the entire audience was laughing and clapping their hands! It's scary to think that these people might well be a representative sample of the ruling Japanese elite.

[1] https://en.wikipedia.org/wiki/Masahiko_Fujiwara

Hi! Author here.

The blog post and source code refer to an older algorithm than the one in the demo. Currently it is using a convolutional network for prediction not dissimilar to [1] or [2]. I'm writing a paper about it, if there is interest I'll put it online :) .

Thank you for the feedback on incorrectly identified characters! The algorithm is not quite perfect - 70% accuracy on a randomly chosen test set.

[1]: http://arxiv.org/abs/1409.4842 [2]: http://arxiv.org/abs/1409.1556

I am not a physicist, but couldn't this be used to transmit information faster than light? Let's say you have object A attracting object B through gravitation, if you could somehow control the inertial movement of object A to encode some information, couldn't this information be retrieved instantaneously at point B by measuring the direction of the gravitational vector pointing towards A ?

I expect the answer to be something along the line of "you can't change the inertial movement of A faster than light would take to go from A to B", but I have no idea why this would be the case. If there's a physicist nearby I'd love an explanation :) .