I.e. do you plan to do the work to align the embeddings of different types of media so that the contents are somewhat similar and e.g. an audio recording of a snippet gets a similar embedding to the equivalent text and a somewhat similar embedding to a picture that's being described?
In the future I think we'll try to embed into a single space on a best-effort basis, assuming we can find the engineering resources. It will be really hard for some data types, but for the big ones like text/image/audio it isn't that hard, and will probably be valuable to people.
In my mind "embedding" carries the connotation that you're moving into a smaller space that's easier to work with, and where things which are similar in some way are near each other.
https://developers.google.com/machine-learning/crash-course/...
https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduc...
When I was learning about SVMs way back when, we said "non-linear projection" instead of "embedding".
So Projection sounds closer to what you describe. https://en.wikipedia.org/wiki/Projection_(mathematics)
So my original comment is not quite right either. The embedding is a structure preserving map. A projection isn't necessarily structure preserving. ''' To be an embedding, such a mapping must preserve order "both ways": ''' http://mathworld.wolfram.com/Embedding.html
I think that a projection from p dimensions to q<p dimensions isn't structure preserving.
At this point a mathematician may be able to explain more.
So, I don't know a ton about Word2Vec which probably doesn't help, but I do understand that it makes tasks, like NLP, much easier since you're going from this massive space (the english language) into a smaller embedding that you learn.
That being said, how are you embedding images? Is it based on how similar they are, if so what does "similarity" mean? Also, what dataset was leveraged?
Any more info on how you do this task would be awesome :).
We're embedding images by feeding them through a deep neural net and using the activations of an intermediate layer as an embedding.
You can read https://arxiv.org/abs/1403.6382 to learn more about this technique if you're interested.
Our launch model is trained on ImageNet, which has enough variety that it usually generalizes well even when your dataset is very dissimilar to the input distribution. We're planning to train on a wider variety of image data in the future, but we wanted to get something into people's hands quickly.
In particular, semantically similar words are close to each other after embedding, so the space ends up with semantically meaningful clusters.
Our embeddings have the same property. If you embed two similar images, they'll end up closer to each other than two dissimilar images. (Where "similar" depends on the training details, but that's true for word2vec as well.)
If you're doing clustering or instance retrieval, you probably want to PCA the number of dimensions down to 200 or so. (In fact, we do this in the tutorial at https://www.basilica.ai/tutorials/how-to-train-an-image-mode... .)
If you're training a big regression, you'll probably get better results with the larger embedding.
We decided to err on the side of making the embeddings too big, because it's very easy to reduce the number of dimensions on the user's end, and impossible to increase it.
I did a lot of the ML work for this. Let me know if you have any questions.
The title might be a little ambitious since we only have two embeddings right now, but it really is our goal to have embeddings for everything. You can see some of our upcoming embeddings at https://www.basilica.ai/available-embeddings/.
We basically want to do for these other datatypes what word2vec did for NLP. We want to turn getting good results with images, audio, etc. from a hard research problem into something you can do on your laptop with scikit.