News for hackers. Stuff that matters.
Re:The Super Effectiveness of Pokémon Embeddings Using Only ... by refulgentis on Sunday June 30, 2024 @10:55PM (#40842162)
To some approximation, if you understood what BERT was at the time it was released, you'd consider it the first modern LLM. GPT-1 was OpenAI's BERT.

Timeline would be viewed as:

2017: transformers

2018: bert

2018: gpt-1

2019: gpt-2

2020: gpt-3

2022: gpt-3.5 (chatgpt)

Re:The Super Effectiveness of Pokémon Embeddings Using Only ... by rolisz on Monday July 01, 2024 @02:39AM (#40843180)
BERT was not large (it was under a billion parameters). And it wasn't an autoregressive language model like GPT
Re:The Super Effectiveness of Pokémon Embeddings Using Only ... by refulgentis on Monday July 01, 2024 @03:11AM (#40843318)
Relative to today's models, it is small.

For the time, it was large.

Re: that it's not auto regressive, that's correct.

Things built on eachother smoothly. Transformers to BERT to GPT.