This is essentially the reason that differentiable methods alone don't work that well for NLP. Language is a discrete arbitrary symbol system, so small changes may have large effects at many levels. ("Dog bites man" is not like "Man bites dog", discrete is not like discreet), etc.
Re:Voyages in sentence space
by OscarCunningham on Friday March 02, 2018 @06:55AM
(#16500769)
I guess that can happen sometimes in actual spaces too. The shortest path from a point near the north pole to the south pole can change dramatically if you slightly alter its endpoints.
Given a high-dimensional space it could be that most possible paths behave in this weird way.
Re:Voyages in sentence space
by jimmytidey on Friday March 02, 2018 @07:52AM
(#16501004)
How useful is the spatial analogy if the space in question defies all our spatial intuitions?
I can see it has mathematical meaning, but isn't the whole post about having intuitive understanding of sentence space?
I think it's less about intuitive understanding, and more about having a flat, numerical representation of a sentence that (qualitatively) satisfies some properties.
Re:Voyages in sentence space
by TuringTest on Friday March 02, 2018 @06:27AM
(#16500641)
Ok, I've played with the "Look around" example and it does behave as a space, with the "Closer to home" option producing somewhat similar results to the starting point, and the slider producing concepts that look farther away from it.
Maybe the "Gimme a gradient" option step is too large for sentences that are far away, and an option to produce more steps, closer between them, would produce more intuitive results?
I feel like the "middle" sentence is not intuitively what you'd normally think of something in between. Even for sentences that are quite close. "Eve walked home slowly" -> "Eve walked home quickly" gives a middle result of "As white hand smiled."
Re:Voyages in sentence space
by rockostrich on Friday March 02, 2018 @10:10AM
(#16501980)
Robin is an author that dabbles in machine learning and farming olives for making olive oil. It's not surprising that his examples aren't robust, but I always enjoy reading about his endeavors because I end up in a rabbit hole on whatever subject he's writing about.
> I am waiting at the entrance of the fantasy kingdom.
...
> I have a revelation about reality.
Main problem is, it doesn't look at all robust. A small change in a single word at the start or end sentence produces a whole different result. That doesn't look like a "space" at all, as insights about closeness do not work in it.