Re:The Super Effectiveness of Pokémon Embeddings Using Only ...
by axpy906 on Sunday June 30, 2024 @02:05PM
(#40838804)
Nice article. I remember the original work. Can you elaborate on this one Max?
> Even if the generative AI industry crashes
[ Parent
| on HN ]
Re:The Super Effectiveness of Pokémon Embeddings Using Only ...
by minimaxir on Sunday June 30, 2024 @02:30PM
(#40838998)
It's a note that embeddings R&D is orthogonal to whatever happens with generative AI even though both involve LLMs.
I'm not saying that generative AI will crash but if it's indeed at the top of the S-curve there could be issues, notwithstanding the cost and legal issues that are only increasing.
Re:The Super Effectiveness of Pokémon Embeddings Using Only ...
by qeternity on Sunday June 30, 2024 @03:50PM
(#40839760)
While there is no real definition of LLM I’m not sure I would say both involve LLMs. There is a trend towards using the hidden state of an LLM as an embedding but this is relatively recent, and overkill for most use-cases. Plenty of embedding models are not large, and it’s fairly trivial to train a small domain-specific embedding model that has incredible utility.
Re:The Super Effectiveness of Pokémon Embeddings Using Only ...
by refulgentis on Sunday June 30, 2024 @10:55PM
(#40842162)
To some approximation, if you understood what BERT was at the time it was released, you'd consider it the first modern LLM. GPT-1 was OpenAI's BERT.
Timeline would be viewed as:
2017: transformers
2018: bert
2018: gpt-1
2019: gpt-2
2020: gpt-3
2022: gpt-3.5 (chatgpt)
Re:The Super Effectiveness of Pokémon Embeddings Using Only ...
by rolisz on Monday July 01, 2024 @02:39AM
(#40843180)
BERT was not large (it was under a billion parameters). And it wasn't an autoregressive language model like GPT
Re:The Super Effectiveness of Pokémon Embeddings Using Only ...
by refulgentis on Monday July 01, 2024 @03:11AM
(#40843318)
Relative to today's models, it is small.
For the time, it was large.
Re: that it's not auto regressive, that's correct.
Things built on eachother smoothly. Transformers to BERT to GPT.
Re:The Super Effectiveness of Pokémon Embeddings Using Only ...
by pqdbr on Sunday June 30, 2024 @02:26PM
(#40838962)
I think the author is implying that even if you can't extract real world value from generative AI, the current AI hype has evolved embeddings to a point they can provide real world value to a lot of projects (like the semantic search demonstrated in the article, where no generative AI was used).