We're already there. Not a single person is alive today that can fully comprehend any production deployment, from the atoms in the server room all the way up to the atoms projecting rendered browser output to an amazon shopper's eyeballs. Any even this seems like an arbitrary place to draw the limits of our infrastructure. But nevermind the blank spots in physics and psychology; even if we stick just to the server processes from conventional OS and chips, hardware and firmware, data and databases, there is nobody alive for whom all of this is a white box.
What I'm getting at is, what if we get to the point where at some levels of the stack there are literally no people who understand that level? If your company's prod infrastructure is miles and miles of AI written and maintained shell scripts, or config files for tools that haven't been worked on for decades because AI-generated boilerplate is cheaper than adding new features? To the extent that even if you hire the right people they can't reverse engineer the stuff and even if they could the cost of making changes to it without AI is prohibitive, because it's all so low level? It's a weird thing to contemplate but seems like one of the more plausible scenarios in which society becomes as dependent on AI as we are today on electricity.
And one of the reasons why we have such an excess of configuration - which includes general-purpose programmability itself - is because we're trying to late bind the entire application.
So the question of debugging is more one of "yes, we're going to replace a panoply of hand-written abstractions with an LLM-generated solution" than it is about the LLM working in an unknowable black-box paradigm. What you're getting is a way to retarget the domain and precision of the solution:
1. Act as an oracle that answers questions directly. This works fine if your goal is to restate the form of a simple dataset. But the black box effect becomes too large really quickly.
2. Assisted code generation. This can write algorithmic boilerplate for you. But it's still relying on the framing of an existing protocol - "in language X, do Y". We got past our amazement at the oracle but are still mostly looking at this step, where it still looks like a human organization is needed to write a software stack.
3. Assisted data model and protocol design. LLMs are protocol geniuses - if you tell it of a new language and its rules, and give it some examples and logical feedback, they will happily go up and down through the abstraction stack as necessary to restate things in that language. This lets you bound the kinds of solutions that the LLM could generate later by the logic of the defined interface. This also aids the human by enabling all solutions to be defined legibly.
Right now we tend to avoid original protocol design because it's human-expensive and so we end up with inertia towards standardized designs, but the LLM does not care. You could tell it to regenerate all the necessary dependencies for your application, and if the system fails "somewhere in the middle" you don't have to debug what it did in more than a cursory sense - you just regenerate the whole system with added precision and legibility until you've drilled down to the specifics. It can even generate documentation!
That is, in the future, protocol-level friction can be trivially overcome, which means the "early binding effect" of hardware and software platforms will evaporate. You just have the data. The data could be stored in an illegible form - but that's a general human-language issue, and not really about the incidental complexities of human-written software.
100%. I fully expect to see the fickle AI spirits of the Warhammer 40K universe retconned into the grandchildren of LLM-driven program synthesis.
Don't worry about Unix skills, worry about your AI skills. The first is only getting more automated and the second is growing and expanding fast.
> running on infrastructure that nobody understands
AI is good at writing docs, summarising, monitoring, rewriting - there is no reason the code base should be opaque. Also, we will be much better at picking up new skills quickly - force of habit, we will be better at grocking things with AI.
The risk here is not that AI can't explain what a particular script does. The risk is that we stop creating simplifying abstractions. Let me try to explain what I'm getting at here because it's a new thought and I'm not doing it well.
Humans have limited and roughly equal mental and physical capacity. Beyond a certain point you have to add more people to get more done. In turn those people need abstractions because they can't grok the whole system (AI assistance or not), and so to organize into teams you need people to abstract away essential complexity. We build on top of a tower of abstractions as a consequence, and to a large degree progress comes via better and higher level abstractions becoming available.
What if this progress stops because AI doesn't need it? The scenario here is that without people really noticing at first the flow of improvements to our tools dry up. PEPs, JEPs, RFCs, new libraries etc stop appearing. Maybe even bugs stop getting fixed if there are workarounds available. After all, why invest time and effort into improving the syntax of Python or JavaScript if the primary user would only be AIs? They don't care, they'll just write things out the inconvenient way without complaining. Why try to make UNIX or the cloud easier to use, if all your configs are written by AI? Why introduce new abstractions at all?
Assume that tech continues to move forward and people's demands continue to get more complex. But now, instead of inventing new tools, we just pay a bit more for AI to brute force through the limitations of old tools. Why would an AI write and maintain a library on GitHub with a nice API when it can just knock off the code it happens to need right there as part of its own task? So code gets more and more verbose over time, until one day the number of missing abstractions becomes so large that the code is incomprehensible. And asking AI to explain won't help either because at best you'd get a book sized explanation. There are no teams, no projects, no abstractions. It's all just an endless jungle of boilerplate and copy/paste coding.
Yes, but the nature of the problem changes a little. Because LLMs are not guaranteed to produce correct results, this might incentivize configuration and programming languages where you can provide a separate specification, possibly generated by another LLM, against which the program or configuration is checked against. This then potentially saves the user a lot of iteration to correct simple mistakes.
Another aspect is that simpler systems/languages are also simpler for LLMs to understand, so an LLM generating a program in a simpler, more expressive programming language has a higher chance of being correct.
I wonder to what extent AI is going to kill comprehensibility of the infrastructure. Over the weekend I did a bit of hacking with a friend. He wants to move out of AWS to Hetzner to save money, but wasn't experienced with Linux sysadmin so I was teaching him some stuff, and simultaneously pitching a little prototype I made of a server config tool. It's sort of like a higher level Ansible: you specify what packages and containers you want, the URLs they should be served on, and it goes ahead and configures unattended upgrades, Docker and Traefik to deliver that outcome. The idea was to extend it to configuring backups with BorgBackup and other common sysadmin-y tasks.
He was enthused! The next day though, things changed. He'd signed up for GPT-4 and it'd just blasted out a pile of Ansible configs for him. He doesn't know Ansible but it didn't matter, he just iterated with the AI a few times and now the problem is solved.
This makes me wonder if there's much point anymore in improving the usability of systems software (programming languages, databases, operating systems, clouds, anything driven by textual configuration). A basic assumption that underlies making better tools is that user's time is valuable, that intuitive and simple systems therefore have value. But the AI's time isn't valuable. It can read some poorly written docs, then the source code of the system, then people getting frustrated on StackOverflow, synthesize all that together and spit out whatever pile of inscrutable config files are needed and it can do so within seconds.
Given this experience, if we extrapolate it forwards, then maybe within a couple of decades many organizations will be running on infrastructure that nobody understands at all. You get cases today where maybe individual programs or workflows are only understood by one guy, but those are extremes and it's understood that it's a bad situation. Maybe in future it'll be normal and to do anything companies will have to ask an AI to do it for them. We already see skills atrophy around things like UNIX sysadmin because nowadays everyone learns AWS instead, which is one reason they can charge so much money for it, so I think as people retire knowledge of how to run Linux systems will steadily disappear. But conceivably people will stop learning the cloud too, and then to do anything with servers it'll be AI or the highway.
You can also apply this to programming languages. Why engage in a high effort project like inventing a Java or Rust if an AI can spit out correct C++ for you, given enough tries at writing tests? Does the motivation to produce anything beyond incremental improvements to existing tools disappear now?
I keep flipping back and forth on the answer to this. On one hand, it seems pretty pointless to further develop this prototype now. Maybe Ansible or AI written shell scripts is the end of the road for making Linux easier to use. On the other hand, AI is remarkably human like. It can also make mistakes, and also benefits from well written docs, simple interfaces and good error messages. So maybe usability still has a purpose, albeit maybe we now need to master a new field of "usability studies for AI".