Re:From Deep to Long Learning
by Herval_freire on Sunday April 09, 2023 @10:59AM
(#35503347)
But this research came out mere weeks after the release of gpt-4. That is in itself rapid. If small incremental changes like this continue on a sort of monthly basis, the trendline points towards something that's not a dead end. That's my view of it.
As with most technology there isn't necessarily always a constant influx of inflection points and paradigm shifts. Improvement will likely creep up on us incrementally. Suddenly one day its clearly more intelligent then a human and we can't point to when it happened.
GPT-4's release doesn't seem like the relevant time marker, since nothing in the article builds on it or depends on it. The paper for H3 was submitted in December 2022.
The pace the last few years definitely seems rapid, just don't want there to be a false impression.
The researchers are also, excited.
We’re especially motivated by applications that could benefit from longer-sequence models – high-resolution imaging, new modalities of data, language models that can read entire books. Imagine giving a language model an entire book and having it summarize the plot, or conditioning a code generation model on all the code you’ve ever written. The possibilities are wild – and we’re excited.