It feels reasonable, consistent to see it as another "old man yells at the skies" scenario, but I do think it's unprecedented for a machine to automate thought itself on an unbounded domain and with such unreliability. We know calculators made people worse at mental math, but at least calculators don't give you off-by-one errors 40–60% of the time with no method of verification.
The reason why we haven't lost literacy to Speakwrites and screen readers is because they required more time and effort than doing it yourself. With AI, the supposed time savings are obvious: you don't put hours into reading the source to write an essay, you just ask ChatGPT, you don't learn programming fundamentals, you just ask for a script that does X, Y, and Z, etc... It feels like a good choice, but you're permanently crippling you education, both in a structured course and in the wild, and the supposed oracle is a slot machine, costing you $avg_tokens*$model_rate a pull. The poor news is slot machines sell.
There is definitely a way to abuse AI in programming, but it doesn’t seem to be very compelling and I don’t think it will get people who do that very far (eg relying on intel sense to save on typing rather than just learning how to type).
ChatGPT is a great writing tool if you already know how to write. You can curate and modify on top of it, allowing you to write your paper faster with the same amount of quality. But again people just using it to write essays or paper without knowing how to write themselves aren’t going to get good results.
Professional tools, from music notation and art to typesetting and programming, are about translating an image inside your mind into something physical. When you know what you're doing, the lack of an interpretable mapping between prompt and generation means you spend more time trying to describe what you want to write instead of just writing it. I'd be much happier with code-generation if it could take a formal specification and either return an error or something that provably implements it. Maybe interpretability research will one day change that, but as they are now, they're simply not tunable or reliable enough to be used as tools. And yes, prompting doesn't count when they increasingly disregard your instructions.
There are many valid uses: I have a tiny WolframAlpha-like script that lets me type some basic computations and the LLM translates that to Python. I sometimes use LLM completions to get some inspiration when writing prose—even though I usually discard them, they still help me think. They can often act as better grammar checkers than LanguageTool, and they make a nice companion to smaller translation models, having their own quirks.
But none of this needs these larger and larger models; I haven't yet tried, but I think fine-tuning some mid-size open-weights LLMs will yield similar or better results. The industry sold the public AGI, not better auto-complete, a fuzzy parser, or a smarter translator, and now they're burning growing piles of money on a saturated research direction to maintain the delusion singularity is 5 months away.