Re:Anyone else witnessing a panic inside NLP orgs of big tec...
by yunyu on Thursday March 16, 2023 @06:13PM
(#35189544)
BERT can process 512 tokens. LLAMA and FLAN-UL2 can process 2048 tokens. GPT-4 can process 32768 tokens, and is much better at ignoring irrelevant context.
These general models can be fine tuned with domain specific data with a very small number of samples, and have surprisingly good transfer performance (beating classical models). New research like LORA/PEFT are making things like continuous finetuning possible. Statistical models also do a much better job at translating sentences to formal structure than the old ways ever did – so I wouldn't necessarily view those fields are disconnected.
I agree with the general sentiment, there are still major issues with the newer generation of models and things aren't fully cracked yet.
These general models can be fine tuned with domain specific data with a very small number of samples, and have surprisingly good transfer performance (beating classical models). New research like LORA/PEFT are making things like continuous finetuning possible. Statistical models also do a much better job at translating sentences to formal structure than the old ways ever did – so I wouldn't necessarily view those fields are disconnected.
I agree with the general sentiment, there are still major issues with the newer generation of models and things aren't fully cracked yet.