Re:Smart Algorithms beat Hardware Acceleration for Large-Sca...
by thesz on Friday March 06, 2020 @08:36AM
(#22503400)
First, you need to compute these embeddings at least once - sparsity, here you are! Second, these embeddings may be different between tasks and accuracy from their use may differ too.
For example, the embeddings produced from CBOW and skipgram word2vec models are strikingly different in cosine similarity sense - different classes of words are similar in CBOW and skipgram.
Re:Smart Algorithms beat Hardware Acceleration for Large-Sca...
by yvdriess on Friday March 06, 2020 @08:39AM
(#22503418)
So you agree that the problem is fundamentally sparse? Embeddings are used to make sparse (e.g. categorical) data possible on GPUs, and real-world models are limited by how large they can make the embeddings to fit in GPU memory. Embedding lookups is also a compute bottleneck:
An example is facebook DLRM: https://arxiv.org/pdf/1906.00091.pdf
Most real-world models don't use one-hot encodings of words -- they use embeddings instead, which are very dense vector representations of words. Outside of the fact that embeddings don't blow out GPU memory, they're also semantically encoded, so similar words cluster together.