So the strengths and weaknesses quickly can become outdated as the strengths grow and weaknesses diminish.
When the first batch of LLMs people tried in 2023 had a lot of weaknesses. At the end of 2024, we can see increases in performance in speed and the complexity of output. People are creating frameworks on top of the LLMs that further increase their value. We went from thousands of tokens in context to millions of tokens pretty fast.
I can see myself dividing problems up into 4 groups:
1. LLMs currently solve the problem
2. It doesn't solve it now, but we are within a couple iteration of next generation models or frameworks to be able to solve it
3. LLMs are still years off from being able to solve this effectively so wait and implement it when it can.
4. LLMs will never solve this.
It seems that the effort required to make an LLM work robustly within a single context (spreadsheet, worddoc, email, whatever) is so gargantuan (honestly) that the returns or even the initial manpower wouldn't be there. So any new feels more or less like bloat, and if not fully useless, then at least a bit anxiety inducing in that you have no clue how much you can rely on it.
Now if LLMs are just effective as your experience says, they are indeed extremely useful and you absolutely should see if they can help you.
It’s only when you attempt to build a product — and it could be one person writing one Python script — that uses LLMs in an automated way with minimal human input that you really get insights into LLMs’ strengths and their limitations. You realize it could be useful, but you have to sometimes baby it a lot.
How many people get to step two? That’s a select few. Most people are stuck in the dreamy phase of trying out interactive LLMs.
This is a re-occurring issue with all new technology. Heck it happens with new software frameworks.