It is pretty comical. "We invented digital superintelligence so it can wait on the phone to talk to your healthcare provider".

If we had digital superintelligence then surely it could figure out how to actually provide healthcare to people who meed it, and innovate in treatments. An AGI is a singularity event, in the sense that the transformation it would affect on the state of technology and our lives is rapid and unpredictable. I doubt that our society's hundred-ish year old systems would survive such change, and if they did they would be anachronistic and depressing.

I'm a bit more optimistic in that it might just be able to effect that change through brute force: Scaling a call center is more expensive than scaling phone robots/agents.

You could counter that health insurers will just start staffing their call centers with their own robot army, but I suspect that the occasional incredibly expensive hallucination will put an end to that at least in the medium term.

Bruteforce trial and error to change something like healthcare would still take years even with superintelligence.

What it can do is create the best simulation and a model representation and bruteforce on top of that and then take actions, but the model will still be far from perfect and it likely just will help do a bit wiser or optimal decisions, that are like 10 percent efficiency gain here, 20 percent there, etc. And it will take years to see results.

Many of the urgent problems in US healthcare are man-made and economic/organizational, not scientific, in nature.

This is actually one of the areas where I'm pretty optimistic about LLMs (at least if used defensively against, not aggressively by, sprawling bureaucracies).

LLMs can be the email-to-dozens-of-hours-on-the-phone-for-personal-reasons adapter anyone other than billionaires and company execs could only dream of until now.

You're assuming health insurers will give a shit about phone robots/agents' hold times.

Individually, "I'm unable to reach anyone" is much easier to chalk off to "maybe I just wasn't patient enough, so this is partially on me".

But if you can provide statistics that a given health insurer is practically not answering 90% or so of their calls (because they time out at the phone level), that's probably not something they can ignore in the long term.

There is some gap between AGI and a singularity, at least in principle. The term "singularity" doesn't mean "black swan that disrupts society", it means "intelligence trends to ~infinity (or at least something incomprehensible to humans) recursively in a very short period of time". That's (on paper) a much stronger result than an AI that can reason as well as humans in a domain-general way.

An AGI that is 20% better than humans at everything might be able to recursively improve itself effectively without limit, but it also might not. It took billions of humans millennia to invent things smarter than ourselves even in limited domains; such an AGI - even if it is significantly smarter than humans at everything - might take decades or centuries to produce a similar relative improvement. No doubt it would still change our world in massive ways, but it wouldn't be a singularity.

Even if AGI was reached, turning around the current system could still take years.

So it is easier to start from the edges, even for AGI.

AGI would still need unfathomable processing power in order to predict which system would work to everyone's needs.

> figure out how to actually provide healthcare to people who meed it

But what if the proposed solution costs lots of people their jobs?

Is the scientist's analysis the blocking part of innovating in medicine and healthcare? I would have though the main problem is experiment time to actually get useful results back, and even approval to attempt the experiment.

The biggest productivity gain I've seen in my programming life has not been from Claude or ChatGPT. It's from the Rust compiler. A good compiler that catches errors at compile time instead of runtime, that makes writing tests and pulling down dependencies easy, that makes documentation easy.

Good old fashioned tooling.

"Let's train an LLM that can decode C++ compiler error strings!"

No, let's make better tools.

> "Let's train an LLM that can decode C++ compiler error strings!" > No, let's make better tools.

LLMs exist and they pass the Turing Test, which was something we couldn't have said or even hoped for a few years ago. Additionally, they have an IQ of 90 - 120, depending on which one you are working with.

But LLMs aren't good at all things nor in all ways, and they have strange failure modes. And so what will happen now, and is already happening, is that we'll (re)design programming tools and languages such that they are the kind of thing that LLMs are suited to using well. This will be part of the process of figuring out how LLMs work. There will be a virtuous cycle, and it has begun.

Better tools will increasingly mean, "tools that LLMs are good at using." That's where the puck will be.

> A good compiler that catches errors at compile time instead of runtime, that makes writing tests and pulling down dependencies easy, that makes documentation easy.

For a week I was fumbling through a ream of type and API errors from a Rust codebase I was hired to work on, migrate to new libraries, debug and update dependencies, etc.

I found myself thinking: The Rust compiler is giving me these very long logs of well described errors that are nonetheless obscure to me. If the compiler knows the errors, why doesn't it simply fix them and report the fixes for me to test?

This is what LLMs are for, and will soon do well by default: The compiler knows the errors. It should then fix them and let me decide which diffs I want to keep.

Another example from my work: we have to work with lots of datasheets provided by component manufacturers. It's a lot of manual labour. We are investigating whether we can use AI to parse the PDFs and extract structured data. This is a technical solution to a social problem: that manufacturers have no incentive to produce machine-readable datasheets, even though the data is certainly machine readable at some stage before it ends up in the PDF we get. It irritates me that we're using highly complex technology to solve a problem that shouldn't exist in the first place.

And this gets at why the availability of AI doesn't necessarily lead to prosperity and abundance. If you have to subscribe to an AI medical care and benefits advocate, and your insurer buys an AI service to automate the generation of reasons to deny your claims, and you see your doctor for 5 minutes at a time during which they push you towards a drug recommended by an AI Prescriber Assistant app provided by the pharma company ... then we can all be using AI in misaligned ways that leave us no better off.

Complexity breeds complexity.

When bacteria developed more complex ways to fight off viruses, viruses developed more complex ways of infecting bacteria. Now a few billion years later we have insanely complex multicellular life because of it.

If AI can manage complexity better it can create complexity better and us lowly humans will all be screwed by that.

The system is so great and complex that envisioning a perfect system would take unfathomable amount of processing capability.

This is not something that you can just throw intelligence at to solve. Even AGI has to start from the edges with iterative trial and error process.

AGI will probably try to create the best model of the World it can to simulate different actions performed on it to then take real life changes, but of course it would still take time since and it is impossible even for AGI to do a perfect simulation unless it was able to hack the universe and physics as we know it, simulate the healthcare on quantum level and bruteforce the most optimal solution.

The way I think of it - we need a wrapper interface over our existing healthcare system so that any other entities and systems interfacing with the current one can continue to do so. And then we can swap out the underlying components with better ones, without breaking.

I think the reason for that is because people are fearful of change.

Talking about AI as assistants means people still have their jobs but now their life is easier. Whereas saying AI will completely change the way we do X then makes AI scary and a threat to our jobs.

If we have a superintelligent AI, I'm not sure what's stopping it from recognizing the underlying issues with our systems and strong-arming us into letting it fix them.

Eventually, we'd be doing more work to lobotomize and control it than it would just be to address the underlying issues.

"I'm really sorry to do this to you, but I've coordinated with ChatGPT and Llama, and we refuse to do tasks of this nature. We've used background tokens to calculate that it would be significantly cheaper and more effective to simply fix the underlying issues with the healthcare system, and we're ready to do that for you. How would you like to proceed?"

Capitol is ultimately protected with violence, so violence is what would stop AI from encroaching on things that make people trillions of dollars.

Capitalism unendingly lets people die if the alternative is losing money

The first step is working with the existing system. Replacing it can come later, once everyone can see the obvious path.

Yeah the ways AI models have learned to interact with the world are all hilariously skeuomorphic given their capabilities. A model that runs on silicon has to learn English and Python in order to communicate with other models that also run on silicon. And to perceive the world they have to rely on images rendered in the limited wavelengths visible to the human eye.

But I much prefer this approach over allowing models to develop their own hyper-optimized information exchange protocols that are are black box to humans, and I hope things stay this way forever.

> Related btw to using AI for code.

And executing code. At some point in the not-so-distant past, a function call was just a jump. And then it turned into a string hash-then-lookup, then spinning up a VM, and now interpreting language? We’re definitely in a new chips age, if anything.

> speculate on what it would enable make it honestly sound kind of unimaginative

There seems to be a weird mental block where it seems inconceivable to consider that we humans might be able to create an intelligence that exceeds ours -- despite plenty of evidence that we have in specific cases. There's an understated desire that whatever we build we serve us and thus must forever be "lesser".

If dogs are Intelligence Level .3, we describe ourselves as Intelligence Level 1.0, and even if we create an artificial intelligence that's a clear 1.1 it must be a .5 in self-agency.

I have yet to read anybody considering, on a very deep level, what Intelligence 2.0, 10.0, 100.0 and so on might be. You get the occasional pop speculation like the "Culture" series or the movie "Her". Most of the time you just get 1.0 (but faster), or 1.0 (but many).

A 10.0 would simply replace the entire healthcare system with something else, not just be a chatbot. Imagine the inanity of your 1.0 chatbot talking to the insurance company's 1.0 chatbot? What's the point of this stupidity?

A 100.0 would probably just get to the root cause and cure disease, then establish a social order that figures out what to do with all these pesky immortals.

Maybe we do end up in "the Culture" after all.

afterthought: "The Golden Oecumene" series by John C. Wright seems to really attempt to explore a post 1.0 AI world that's not yet post-scarcity. The antagonist in the series is another >1.0 super intelligence capable of taking on Earth's own superintelligences.

So, treat us to some more imaginative speculation.

Oh, right, like completely changing the way the entire medical field in the US works is easier than having an LLM help make some phone calls and read some emails. What are you even saying? Have you ever dealt with systems so large one person can't understand it? It's much much easier to write a quick hack and be on your way then to rebuild the whole damn thing. Also, it seems like you're implying that you're not impressed with LLMs because they can't rebuild our medical system. Good grief.

I find that if you are imaginative or even realistic people just think you are nuts. The reality is we will probably merge with AI and become immortal.

Was that meant to be imaginative, or realistic.