Re:The Super Effectiveness of Pokémon Embeddings Using Only ...
by qeternity on Sunday June 30, 2024 @03:50PM
(#40839760)
While there is no real definition of LLM I’m not sure I would say both involve LLMs. There is a trend towards using the hidden state of an LLM as an embedding but this is relatively recent, and overkill for most use-cases. Plenty of embedding models are not large, and it’s fairly trivial to train a small domain-specific embedding model that has incredible utility.
Re:The Super Effectiveness of Pokémon Embeddings Using Only ...
by refulgentis on Sunday June 30, 2024 @10:55PM
(#40842162)
To some approximation, if you understood what BERT was at the time it was released, you'd consider it the first modern LLM. GPT-1 was OpenAI's BERT.
Timeline would be viewed as:
2017: transformers
2018: bert
2018: gpt-1
2019: gpt-2
2020: gpt-3
2022: gpt-3.5 (chatgpt)
Re:The Super Effectiveness of Pokémon Embeddings Using Only ...
by rolisz on Monday July 01, 2024 @02:39AM
(#40843180)
BERT was not large (it was under a billion parameters). And it wasn't an autoregressive language model like GPT
Re:The Super Effectiveness of Pokémon Embeddings Using Only ...
by refulgentis on Monday July 01, 2024 @03:11AM
(#40843318)
Relative to today's models, it is small.
For the time, it was large.
Re: that it's not auto regressive, that's correct.
Things built on eachother smoothly. Transformers to BERT to GPT.
I'm not saying that generative AI will crash but if it's indeed at the top of the S-curve there could be issues, notwithstanding the cost and legal issues that are only increasing.