You may already know this part, but for anyone who doesn't know: vector space models very often use cosine distance to make comparisons, instead of Euclidean distance. In this model, you can visualize your vectors as points on a unit hypersphere, where the distance between two vectors is how far apart they are on the sphere. x+y finds the point between them, x-y finds the point between x and the antipode of y (or, perhaps more intuitively, pushes x away from y). 2*x doesn't have any additional meaning (it's the same as x). But x+2y is the equivalent of finding a point that is proportionally closer to y than x (I think this is 2 times as close to y than x, but I didn't do the math). edit: to be clear this paragraph is very hand-wavy on the math, and is just meant to create an accurate-enough visual.

My intuition of why things like king + man - woman work is because the points in the vector space model happen to create a well-behaving manifold with smooth meaning changes. It's not very principled, but it does work.

I wrote a series of blog posts with a coworker about doing this with music:

https://tech.iheart.com/mapping-the-world-of-music-using-mac...

https://tech.iheart.com/mapping-the-world-of-music-using-mac...

Instead of mixing nouns and adjectives, we do things like mixing songs and artists and radio stations etc. In the second post we show how Nirvana - Kurt Cobain + Female Vocalist works remarkably well. I've studied empirically why this worked, and the best I could come up with is that the high dimensional space we created had a very dense set of points in the region of popular western music that led to a smooth manifold.

Those posts are really interesting.

I address this point. For scalar product, well, these word vectors are defined such that their scalar product approximate the pointwise mutual information between a pair of words (or sometimes other quantity, depending on the actual algorithm).

For differences - see this section: http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html#....

I mention also that multiplying a word by a factor (for PMI compression) results in a word of similar meaning, just being more characteristic (bear in mind that for other models in can be related e.g. to word frequency or other properties).

But you are right, that there are some problems with linear structure. Some of them were brought to me be Omer Levy (a researcher in this subject). I think an article that summarises it the best (or rather: shows empirically that it does not always work as intended) is:

- Tal Linzen, Issues in evaluating semantic spaces using word analogies, https://arxiv.org/abs/1606.07736

Right, I don't deny that you get something sort of vector-ish and some operations that make intuitive sense some of the time. That's cool enough on its own! There's no need to pretend that we've actually constructed a vector space. Just say that we've arranged words on a hypersphere with an inferred distance metric and call it a day.

(Also, by "scalar product" I meant "scalar multiplication"--the product of a scalar and a vector, not the dot product. It's pretty clear how to make some sense out of the dot product, but it's pretty hard to make consistent sense out of scalar multiplication. Apologies for being unclear.)

There isn't a word at every point in the space. Is that your objection to calling it a vector space? I think of every point in space as representing subtle gradiations of concepts, only some of which are instantiated by words.

In any case "vector-ish" is an appropriate word here. As with logistic regression for complex problems:

"Essentially, all models are wrong, but some are useful." - George Box

It seems to behave a little like positions, then. London makes sense. Paris makes sense. London - Paris makes sense. But 2*London does not.

> London - Paris makes sense. But 2London does not.

I guess you know this, but for others: London-Paris makes sense, but has a different type to London and Paris (it's a vector, not a position) and while 2London doesn't make sense, 2(London-Paris) does make sense (it's a vector with the same direction but twice the length).

Such a system, with two distinct types -- positions and vectors, with vectors being the differences between positions -- is called an affine space. You can identify positions with vectors by picking a distinguished origin, but then you don't get the type-safety that forbids ridiculous expressions like 2London.

I thought it was putting words on the surface of a sphere and just doing math with cosine distance?

That's fine, but then you shouldn't pretend that you're in a vector space. Words have meanings, and mathematical concepts have precise definitions.

Constraining points to a hypersphere surface only reduces dimensionality by one dimension. For example a sphere in 3-space still has a 2-dimensional surface to play with. That's still a vector space right?

No; it's not closed under linear combinations and it doesn't contain an additive identity.

Well it does if you use the right coordinate system! As the article mentions, most people don't normalize correctly and only get passable answers by luck. You can add locations on the earth's surface if you pick a point to be the origin.

Then you really shouldn't look too closely at physics.

What is a secondmeter? A voltvoltgram?

I don't follow the point you're making. A volt plus a meter is just that, a volt plus a meter. A king minus a man plus a woman is a nonsense statement. Neat, but not as not exactly illuminating.

I'm glad it can be useful. But, I agree it seems to leave a lot lacking.

A king minus a man is a monarch. Add a woman and you get a queen. I don't see why this is so complicated. It is just like adding or removing adjectives from something.

This is nonsensical if you don't already have some existing notion of a relationship. These are vectors, not concepts. There a loads of operations that don't have any intuitive notion, yet are well defined in the context of vectors and most likely yield useless results.

Well isn't that true of all abstractions? They give us tools that we couldn't use before, and some of those tools don't make sense. I mean, in real physics it doesn't make sense to add different dimensions, eg time and distance, but math lets us do that, because they are 'just numbers'. We need to know that dimensions can only be multiplied and divided, and dimensional analysis needs to be done afterwards to check the unit of the result.

However, something like the dot-product here does make sense, since you can use it to determine similarities of vectors.

>This is nonsensical if you don't already have some existing notion of a relationship.

The engine does have an emergent notion of the relationship, which is the whole point.

That doesn't follow in any addition sense. A king minus man can just be a dead king. Or a eunuch. Or, a lonely king.

Similarly, add a woman to a person, and you just have two people. One being a woman.

I get that there is an answer that seems fun... But there is not a deep meaning to the math.

This is like the games where "send + more = money". Fun. But is there really something illuminating?

Stop thinking literally, and instead think of the concept. A king is a male monarch. So remove the male, and you have monarch. Add female and you have female monarch = queen.

Similarly with woman + person, the concept of woman is femininity. It makes sense if you think in terms of concepts.

>That doesn't follow in any addition sense. A king minus man can just be a dead king. Or a eunuch. Or, a lonely king.

Yes, and some of those meanings can be closer to the common understanding of reality or the weight of each notion or its probability than others.

Which is also why we can solve riddles and don't get lost in their infinite similar possibilities.

>A king minus a man plus a woman is a nonsense statement.

It's not supposed to be english, it's a query language.

King - man + woman actually is illuminating. Your 6 year old son might even say that's what a queen is. There is useful math here.

I call shenanigans. Find me a place where that algebra, which lacks standard rules of common algebra, makes sense.

Consider, king + woman = queen + man. Which looks neat, but is not a universal truth. It could be concubine, for example.

So, is queen + man also concubine?

Again. I'm glad this works for some things. But really just shows which words are often used together. It does not show any good rationale for their meanings. Unlike math, where 1 + 1 equals 2. Possibly in different encodings. But not just from convention of often being used together.

The answers you are looking for are Nobel-worthy. I have not formally studied linguistics, but you might wanna start there. Regarding the king equation, I think "king" is actually a vector in this context, as with the other objects. If so, then it is using the vector space formalism.

That said, I definitely agree with you. An English speaker may find any of these reasonable:

1. King - man = expensive clothing 2. King - man = prince 3. King - man = queen

What is "king"? What is "-"? What is "man"?

If a king is a dressed up wealthy man, and you remove the man, you have wealthy clothes? Or, does removing the man mean degrading the king back into a boy? Or, does removing a man mean adding a woman? Wait -- what if a king is more than a dressed up wealthy man? Should we include his home? Do we need to subtract the home? How do you subtract a home? Is the king minus a man a prince if the king was a beggar when he was young? ... death of the universe ...

Like you said, there's a combinatoric explosion here. Maybe this example is akin to trying to model each and every trajectory of all 10^23 particles in a gas. It looks like these scientists are stepping back, and looking at the big picture, instead, trying to find something more akin to PV = NkT

>It could be concubine, for example.

Precisely. Models which use this space do not propose strong equality (==). Rather, they would output a series of probabilities, and choose the most likely. Stating king + woman = queen + man is somewhat disingenuous; what should be said (mathematically) is something like the following: the word lying closest to the vector vec('king') + vec('woman') - vec('man') is 'queen'.

To suggest that a NN can't learn something about the meaning of words from a large corpus of text is unsubstantiated, I believe. The statement above suggests they do, I would say. I would not be too surprised if a sufficiently complex NN could 'learn' the concept of gender with a corpus of English text to a decently high degree of accuracy, simply based on vestigial features left from French and Old English.

It makes more sense to sort of factorise each thing into its different components. King is a masculine monarch. Man is a masculine person. Queen and woman are the feminine equivalents.

So king + woman = queen + man is better described as:

Masculine monarch feminine person = feminine monarch masculine person

A bit of reordering of adjectives and it is exactly the same. Even monarch is the wrong word, because you seem to be getting hung up on nouns, when these are all actually a bunch of chained adjectives. Perhaps "regality + nobility + rulery". English is a bad language to describe this, because we tend to noun and verb our adjectives regularly.

>Consider, king + woman = queen + man. Which looks neat, but is not a universal truth. It could be concubine, for example.

That's totally fime, because actual words don't define universal truths either.

Queen could be a band, a transvestite, an actual queen, and several other things besides.

That's the deficiency of AI right now... pattern matching is only one aspect of intelligence that's fairly narrow and only moderately sophisticated.

It's still useful -- you can classify millions of pictures in a meaningful way much faster than before.

I was not claiming against its usefulness. I merely question if it truly has meaning.

That is, x+y has meaning and use in most maths. Here, it seems primarily use.

Well, if I had to give it any meaning I would probably choose the word with the smaller distance from the one that was computed (provided that a sensed scalar product exists here). Closure under linear combinations is not a problem if you separate the "word vector" from the actual word. With the same reasoning as above 0.999999king would still be king.

The fundamental problem is that linearity is probably not that accurate in representing words relationships, but I don't think this is the goal.

> What is 2x"king"?

War? Murder?

:)

Diety