Any of these can lead to dystopia.
When we use computer vision to identify benign or malignant tumors, we might cost someone a job, but at least what that model was trained on can be known and we are not leaking personal data in the model and probably there is no violation of licenses or similar to the data that was used to train the model. The model is very limited in output. The consequences are kind of simple to monitor or predict, because the model can only be used in its specialized area of use.
With the new crop of ML models the situation is different. Lawmakers were asleep while these things became popular and there is no knowing, how it will all pan out, because the output is of a more general nature, that can touch any subject, any area of expertise. No one knows what they have been trained on and whether there is anything in that training data, that should not have been used, and there is no simple way (that I know of) to operate out any such data after training the model.
Here's the rub: if my competitors are going to realize "massive productivity gains," where are those gains actually coming from?
Are they coming from the time it takes to type? No, because typing is a small portion of the time we spend as programmers.
Are the gains coming from the time spent on design and architecture? Maybe, but that means my software will have better architecture, and I will have better velocity in the long run. (I have real-world experience with this, by the way; it takes me, on average, less than an hour to fix bugs in my bc nowadays. Architecture matters for long-term velocity.)
Are the gains coming from the time squashing bugs? No, this is where AI would decrease productivity because you have to carefully review outputs to ensure there are not any bugs.
In essence, the productivity gains will only happen the less that my competitors actually review and clean up the code from these tools. The less they do that, the lower the quality. The lower the quality, the slower their long-term velocity will be.
A higher slope makes up for a lot of y-intercept. [1]
[1]: https://gist.github.com/gtallen1187/e83ed02eac6cc8d7e185
I'm just advocating a very cautious middle path. For now, these tools are just code helpers to me (and right now I barely use them for that), but the moment I sense a genuine advantage to using them in my job, and I don't think it undermines my engineering, I think I'll have to engage further. Undermining for me means, reducing software quality, reliability, my own understanding. To the extent that I can work faster though, and even improve on those metrics, so much the better.
The semi-near future to me (5-10 years?) , breathless hype-aside, looks like one where quite a lot of coding is automated with machines able to range over whole codebases and build higher level models of the software, and then getting into architectural system-design suggestions. I also see them linking up with company docs to build up a model of the whole domain. This doesn't need AGI, it's just an extension of what we have now, and I can't see it being anything other than a game changer.
Forget ChatGPT, imagine your own private LLM, trained on your code and business, with all the weird glitches these prototypes have currently largely gone. You, the expert architect, are still needed, but you've now added a superpower to your tool chain.
> This doesn't need AGI
This is where we disagree.
Code is so complex that I believe, strongly, that anything less than AGI will not be good enough. Notice I said "good enough". Sure, something less might be able to do something, but not good enough to make it a part of my toolkit.
Also, code is Turing-complete. These LLM's are not Turing-complete. How could something that is not Turing-complete hope to "understand" (for some definition) something that is?
Anyway, all of that aside my point is this. AGI-aside, the current tools, scaled up and improved, will have a big impact on productivity eventually. When I think of the toil so much development involves now, especially for your average mid-level engineer, a lot of that could easily be eliminated. That's a lot of man hours right there. I can also see it as a knowledge base on steroids. Large orgs are so hampered by fragmented docs, misalignment etc, that this could really help to build business and domain insights.
I see any number of transformative business /dev products to start appearing soon. The level of investment, and the technical progress, make this inevitable.
Aren't they? I view them as "language processors", that execute one token (instruction) at a time, and use the token window as their working memory.
Sure, they have a limit to their instruction count (8k/32) before stopping. The working memory is small, but 8k tokens goes a lot longer than 8 kB.
I've seen you state in other comments that LLM's cannot reason. I used to think so too until a couple of weeks ago.
But I had to change my mind. They're not just fancy autocomplete tools any more. They exhibit intelligence in that they can understand a given problem statement, reason about it, and make predictions and recommendations.
In order to be Turing-complete, they would need to have edges between nodes (I don't know what they are called in ML parlance) that can go to previous nodes, as well as edges that are only conditionally taken.
> They exhibit intelligence in that they can understand a given problem statement, reason about it, and make predictions and recommendations.
They exhibit the intelligence of the humans that wrote the material used to train their models, nothing more. They are borrowed intelligence.
I just tried this with all three versions of ChatGPT (3.5 legacy, 3.5, 4), and they all did it correctly.
>In order to be Turing-complete, they would need to have edges between nodes (I don't know what they are called in ML parlance) that can go to previous nodes, as well as edges that are only conditionally taken.
If you look at a neural network at the lowest level, it is trivial to implement a NAND gate with them. It just takes a few nodes. And LLMs have a feedback loop through their context window.
With NAND gates and a feedback loop you can build any CPU.
There are many examples of people asking ChatGPT to act like a CPU and pretend to execute instructions, or like a Python interpreter, or a Dungeon Master and respond to what the human players do.
Now, that doesn't make it an AI in itself. But it shows that there is no fundamental limit to its computational capabilities.
> They exhibit the intelligence of the humans that wrote the material used to train their models, nothing more.
Do you mean that they're just pattern matching input text against a fuzzy database and responding with canned answers?
If so, that's not what's going on in my view. That might have been accurate for the earlier LLMs.
But the latest LLMs appear to have developed internal language processing, a world model, and capabilities to reason about things.
Here's one example:
Prompt:
Please pretend you are a CPU. Execute the following Python program and show what the output would be. Respond only with the outputs, and no comments:
for i in range(5):
if i == 3:
print("Found 3")
else:
print(i)
Response:
0
1
2
Found 3
4
Second example (only works in GPT-4):
Prompt 1:
Please pretend you are a CPU. Execute the following Python program and show what the output would be. Respond only with the outputs, and no comments:
stop = False
while not stop:
user_input = input()
if user_input == "stop":
stop = True
print(user_input)
print("Program is done")
For "input()", stop and ask me for the input string.
If you reach the statement 'print("Program is done")', then say "It looks like we are done!".
Response 1:
input()
Prompt 2:
hello
Response 2:
hello
input()
Prompt 1:
world
Response 2:
world
input()
Prompt 1:
stop
Response 2:
stop
It looks like we are done!I suspect this is one of those tech Pandora's box moments where it's largely irrelevant what actual tech people think about this; executives will be swayed to the "use AI everywhere" side of things purely as a cost-cutting measure. Any productivity gained via the use of AI will be leveraged not to improve overall productivity (in the medium to long term) but instead leveraged against employees to to shed salaries.
On avoiding it, if your competitors are getting massive productivity gains from AI tools, your hand-crafted, artisanal code is going to be expensive and your customers will vote with their wallets.