Why would an untranslatable word take on bizarre values? If there are enough training examples of the untranslatable word, why couldn't it actually have a sensible mapping in the vectorspace?
p.s. thanks for the reference to that thesis. I've been wondering about uncertainty in predictions for a side project to predict athletic performance that I'm working on, so this is very timely.
word2vec is trying to predict the co-occurence of words; if there is nothing translatable about it, then it will be effectively random. When you fit a maximum likelihood to random stuff and force it to attribute pattern to the noise, it's not going to come up with any sensible mapping.
But just because it's "untranslatable" from, say, Chinese to English, doesn't mean that we can't see many Chinese-Chinese co occurrences of the word, so this Chinese word would still have a vector of significance relative to other Chinese words, right?
That is- if you ran word2vec on a Chinese corpus, this word would be represented adequately in the vectorspace.
Wouldn't it then make sense, if you could find a proper mapping from one language's space to the other (I don't know if this part is realistic, but maybe by finding correspondences between already-translated words), that this word would not land in a bizarre spot were it to be expressed as a combination of vectors from the other language?
No. It'll still be mapped onto the word2vec vectors, but it would just take on bizarre, extreme, and highly unstable values due to maximum likelihood training. It'll associate that untranslatable word with all sorts of random other words like 'cheese' simply because 'cheese' happened to be used within a few sentences of the untranslatable, and the mapping will vary unpredictably with different hyperparameters, random seeds, datasets, implementations etc.
This often happens; for example, in a simple logistic regression, suppose you have 10 datapoints and 1 variable that separates them perfectly, what will maximum likelihood do? It won't give you a sensible odds ratio like 10, it'll give you an OR of ∞, because if the OR is ∞, that maximizes the chance of perfect separation since the likelihood of separation is always 1 and any finite OR would imply a slightly smaller likelihood <1. (While a logistic regression using any kind of regularization or Bayesian prior will behave much better.) You can see this in image-classification CNNs when you give them a photo of something not in the training set - they just come up with whatever is the closest match for the flimsiest and most superficial of reasons. Or if you try to use a NN for time-series prediction and extrapolate out the prediction, it'll be an exact extreme trend, because it's not reflecting any of the real uncertainty.
This is why being able to ask a NN about its uncertainty is so useful: the regular NNs will just confidently predict bullshit because that's what you trained them to do. There's some nifty stuff about this in "Uncertainty in Deep Learning" http://mlg.eng.cam.ac.uk/yarin/blog_2248.html , Gal 2016.