I don’t think any company is seriously considering that, having played with chatgpt for a bit. Getting it to generate functional code requires so much guardrailing and validating the output that you basically need the domain knowledge anyhow as well as the ability to code to even use ai well as it is. I don’t believe it will ever get to a state where it can just be trusted to write good code for a complicated purpose without a skilled human shepherding it all along the way. The problems we solve are too bespoke a lot of the times. Its just a problem of how these models are fundamentally constructed: train on data and project new data onto it. Problems arise when you understand training sets are finite and contain things that already exist.

No, it will be more subtle. The features you will get in software products, as a user, will get AI-ified. Mostly for cost reasons. What it means, instead of building a proper feature that works correctly and handles all the edge cases, AI will be invoked to resolve the discrepancies and "make it work". It will mostly work, but will be less reliable and understandable. The end result will be more shifting of burden of errors from the software companies to users.

That is just dystopian enough to be believable.

If I see the crunch at my own corp and of friends. We are at the point we're we just make it work.

Reminds me of this episode of star trek: https://en.wikipedia.org/wiki/When_the_Bough_Breaks_(Star_Tr...

It's about an ancient civilization so advanced they have tools and computers that can create art just from their intent. But they have grown so distant from the inner workings of these tools that they are incapable of modifying or repairing them and their survival is put into jeopardy because of it.

Or The Machine Stops by E. M. Forster, which I think has held up pretty well for something published 114 years ago.

https://en.wikipedia.org/wiki/The_Machine_Stops

If we're on the topic of Star Trek, observe how the crew of Enterprise-D talked with the ship's computer. LLMs are the quality jump in technology that, for the first time, make it possible to create this kind of DWIM interaction, where the computer can actually understand what you mean when you talk to it like to a person, in natural language, instead of a restricted language.

Ironically, later shows dialed down on the smartness of Starfleet computers - from what I heard, TNG level voice interface seemed too advanced to be believable. But it turns out, the unbelievable became reality sooner than anyone expected.

If only reality had given us warp drives instead of talking computers.

For as substandard as most of the season 1 TNG episodes were, there was a lot of interesting ideas sprinkled here and there like the one you mentioned. It's too bad that poor acting and the ridiculously small scale of the conflict in the episode distracted from things like that. They probably should have abandoned the whole "we can't reproduce so let's kidnap children" thing for contrasting that civilization's relationship to technology with that of Earth's. IMO, there's already a moral dilemma in whether a failing technological civilization should be taught how to reproduce by outsiders whose relationship with technology is successful.

Sub what “standard”? I, for one, find the “mixed bag of soapy philosophy” a timeless treat. Not including the later Enterprise and the JJA movies.

I think the standard was set in the last few seasons of TNG and the entirety of DS9. The crew gelled better and the topics were more weighty, yet human. I think probably it was around the time that Picard became Locutus and then the cardassian war

This is us now if you squint.

Survival risk, maybe not, but we are certain to see massive die-off and a decades-long cold start given a perturbation maybe an order larger than the COVID pandemic, when the system is already undergoing unprecedented stress from climate change, as appears likely in the coming century...

COVID was already a lot of disruption to industrial supply chains. A one-two punch like that and we might see some sizeable societal collapse.

The problem is the perverse incentive of the market currents. It doesn't matter how you do it, just make the product work before the competition does (an cheaper). If anything crops up, you'll fix it later. AI promises just that and will act as a negative feedback loop for our understanding of the code. The result is Idiocracy scenario where people are just too dumb to realize that the AI organizing the society is not that intelligent.

Shameless self-plug: I wrote a couple of short stories which explore the idea that society forgets how to do stuff after outsourcing its knowledge to machines

https://ashwinsundar.com/blog/compiled/untitled-part1.html

https://ashwinsundar.com/blog/compiled/automonk.html

I was thinking about https://en.wikipedia.org/wiki/The_Feeling_of_Power

I think you need both!

Our underlying data should still be extremely principled and structured. But I think LLMs enable significantly better human computer interaction, where humans aren't interacting directly with the data in your preferred manner.

For example, I've been parsing sentiment across the internet from the app I run, and using it to crunch it down into a very structured internal tracking database I have. I want to canvass as many places as possible. Using an LLM for that transformation has been stellar, and I'd casually say I'm at around 90% accuracy, which is more than sufficient for my planning & justification needs.

In that sense, it should just be an additional tool in a toolbox, not a platform to bet the house on.

> LLMs enable significantly better human computer interaction, where humans aren't interacting directly with the data in your preferred manner.

It doesn’t explain why a randomized black box is needed. It only shows that we need better tools (i.e. not randomized and open).

I agree completely.

Having LLMs use programming languages/tools/libraries built for humans is the wrong path to fully leverage their potential. They are capable of so much, but trying to use them as a drop-in replacement for humans isn't the best use for them. Frankly, it's impressive it works at all, but I think we will find hard limits on the actual complexity and reliability of the systems they can build in this manner.

We still need intelligent, rigorous system design. LLMs have the strong potential to let system builders operate at a higher level of abstraction, but in my opinion having LLMs write Python or SQL is an evolutionary dead end.

My own work is directly focused on this and what IS the proper place of a LLM in a data processing or knowledge system.