It's like saying a calculator could spontaneously derive its own proof. No it can't, because it has no ability to do that. It physically does not have a way to create new ideas. It cannot reshape its own circuitry like a brain and it does not have spontaneous thoughts.
What makes an idea "novel"? What types of physical processes can create new ideas or have spontaneous thoughts, and which can't? What is "special" about brains, in your view?
I'm looking for some meat on the bone here.
At the time I have to admit I was equally bullishly myopic, the new models seemed insanely cool and the process seemed super-exponential. I predicted AI may replace programming, art and many things within 5 years.
Then I checked in 5 years later - not much really had changed and I revised my predictions way down. Since then I’ve revised them down again actually, as I see them running into scaling issues again, amongst many other issues.
The attitude you take is infectious, I saw it firsthand. But if I look at what concrete is changing, it’s not much.
Humans love watching Chess. It’s exploded in popularity in the last 4 years. No one watches AI chess, though the top players certainly gain in their practice from it. But where’s the revolution?
I know it’s different but I predict the same happens across the “new” areas - image and text generation. They seem amazing, likely beating humans in many ways. But in the end we don’t care much about computers, we care about novelty.
Novelty and humanity drive fashion, music, film, literature… basically all art. It needs to innovate, needs to be extremely personal. It’s not going to disrupt any of them, not even close. But it will be used in the loop.
It’s very distinguishable from a human in that you can trivially make it contradict, say nonsense, and it still makes all sorts of errors especially past a few paragraphs.
Very few people are using it. AI therapy is really a great counterexample - it’s not just not up to the task, also highly dangerous. Last thing anyone needs in therapy is to talk to a computer that has no long term memory and will say whatever you want it to - pure demoralization.
As far as code - also not really compelling. I’ve tried it and don’t use it, because:
1. It’s worse than Google. Google + stack overflow finds 100% of the same answers, but also gives me many alternatives, verifies they are correct, and adds lots of context.
2. It’s incorrect in subtle ways, often.
3. It’s actually actively bad at programming beyond the simplest smallest isolated problems, which are about 1% of the daily tasks of a software dev.
I did say in the loop it has some value, but it’s truly far from an iPhone in terms of value. About 0% rounded of the population is even aware of it, let alone getting value from it. I have no idea what I’d use it for, its proneness to blunders and bullshit is a real dealbreaker for every application besides illustrations and maybe summarization - and the latters been solved forever, and is a marginal industry.
Again there’s a weird need to hype it up it seems, we’re at some peak of the hype curve and people are buying it. I think one reason that happens is because it’s mediocre at a lot of things, so people think it’ll do everything. But it’s got major reasons why it’s mediocre that haven’t changed from gpt3 or earlier.
It’s a cool thing certainly, but Google is like 1000x more valuable and idk if even Google was a revolution or just a somewhat big improvement.
Four fundamental limiting factors: it’s imprecise (can’t rely on it for unsupervised or technical things), it’s average (not very valuable to art or creative innovation), it’s a liar (not great for many things), and it’s a panderer (therapy, etc).
StackOverflow is almost never wrong at the top answer and again has many coherent responses for many use cases with contextual help, where GPT is often subtly incorrect with no real context (precision struggles). The whole iterative thing is maybe good for learning tools. Just not seeing a revolution, yet.
The struggles for SEO are less than those for GPT, both have to deal with spam and weird data, but GPT you’re also out of date, don’t have much explainability, and basically trust one giant black box of bias that requires higher input-foo whereas Google is also a giant black box of bias but at least you land at SO or Reddit or something where things are concrete and easy to understand.
Again, if this was some iPhone level revolution I’d have no comparable. It’d be way better at something clearly. But it’s not even clearly equal to any of the examples, and imo worse than basically all except “draw me a dragon in a tutu”, caricature artists should be worried!
On second thought, caricature artists are pretty safe since much of that experience is the whole human drawing you thing and also the generative models don’t really do caricature well as of now.
The current solution of 500b parameters or whatever is literally just a lookup table of existing data. They could replace it with an SQLite database with the same amount of data and get the same solution.
There is no thought, there is no reasoning. It can't come up with new ways of doing anything, it just regurgitates based on its pre-programmed ruleset. That's why self driving doesn't work - it doesn't know what a fucking car is. Ask it what a road is or a car and it has no idea, it just has a million miles of road in its database. Ask it how to drive on a country lane and it crashes because that's not in the database. There's nothing there.
I'm pretty sure this is completely false?
ML papers always try to find the most similar examples in the training set, and rarely find a close match.
Sure, there are some very vague similarities, but they are also incredibly different.
A bike and a car both take you places. But the kinds of places they can take you and how effectively they do so differs vastly.
Likewise, both lookup tables and LLMs take an input and then spit something out. But the similarities basically stop there.
What I find amazing is when I ask it to do something like "Write a TV Script about Malcom Saving Kaylee from Aliens" - and then it fills in all the details of the Serenity, places Kaylee in the engine bay , has Mal firing off his pistols, and then retreating back to the Bridge".
Then, I simply type, "Add Elves" - and it rewrites the scenario, with the Elves properly using bows to fight off the aliens, and (this blows my mind), when they retreat to the bridge and Kaylee is worried about the engine, the Elves coherently say that it's okay, that they will use Magic to fix the engine and make it faster - And everything is written far more coherently than I ever would and makes absolute sense.
I get that it's a LLM - but damn, it's been blowing my mind for over a month, and I'm still astonished by it every day. I'm starting to wonder if my own intellect is anything more than an LLM connected to emotions/sense-organs.
Looking at LLMs is also not seeing the forest of Tacotron, neural vocoders, acoustic models, Stable Diffusion, NeRFs...
This is snowballing so fast.