News for hackers. Stuff that matters.
Re:Do Large Language Models learn world models or just surfa... by r_hoods_ghost on Sunday January 22, 2023 @06:52AM (#34476482)
But we now know that there is a smaller box within the bigger black box and have empirical data to back up that assertion, which is a step up from where we were previously. That LLMs have internal models and use them to make predictions means that we now know what level of abstraction we should be talking at and narrows down the exploration space within which to make investigations. Now we know that we can learn more about how LLMs work by investigating how such internal models arise. This sort of result makes LLMs slightly less black boxy than they were previously.
Re:Do Large Language Models learn world models or just surfa... by miohtama on Sunday January 22, 2023 @07:05AM (#34476554)
Because the black box is shrinking, does it mean we might ge able to get similar quality results with smaller models?
Re:Do Large Language Models learn world models or just surfa... by wongarsu on Sunday January 22, 2023 @08:17AM (#34477055)
Eventually better understanding of the mechanisms should allow us to make better models. Right now OpenAI is still exploring the boundaries of "how much does it get better if we just make it bigger", but that is just the first phase of making LLMs. Remember that transformer models are basically brand new, having been invented just six years ago. There are bound to be lots of improvements left that will later be described as low-hanging fruit.