The main issue with current LLMs is that they're autoregressive, i.e. you feed in the whole previous text to generate the next token, which seems extremely wasteful.

Here are some recent slides by LeCun on that topic: https://drive.google.com/file/d/1Ymx_LCVzy7vZXalrVHPXjX9qbpd...