The problem with LLMs as interoperability is they only work sub 100% of the time. Yes they help but the point of the article is what if we spent 100billion on APIs? We absolutely could build something way more interoperable and that’s 100% accurate.

I think about code generation in this space a lot because I’ve been writing Gleam. The LSP code actions are incredible. There’s no “oh sorry I meant to do it the other way” you get with LLMs because everything is strongly typed. What if we spent 100billion on a programming language?

We’ve now spent many hundreds of billions on tools which are powerful but we’ve also chosen to ignore many other ways to spend that money.

If you gave me $100 billion to spend on API interoperability, knowing what I know today, I would spend that money inventing LLMs.

For $100 billion you could get public standards for APIs of all kinds implemented. I don’t think people understand just how much money that is. We’re talking solve extreme hunger and create international api standards afterwards money.

Having lurked around in the edges of various standardization processes for 20+ years I don't think this is a problem that gets fixed by money.

You can spend an enormous amount of money building out a standard like SOAP which might then turn out not to have nearly as much long-running as the specification authors expected.

That’s totally fair. Money though is a representation of desire and the reality is people don’t have interest in solving these problems. We live in a society where there’s much more interest in creating something that might be god than solving other problems. And that’s really the main point of the article.

But also even if the W3C spent $10m a year for the 10 years SOAP was being actively developed according to Wikipedia that would still be 1/1000 of the 100billion we’re talking about. So we really have no idea what this sort of money could do if mobilized in other ways.

Yeah. Much of that money is going to physically building data centers, in the middle of an affordable housing crisis. "Look, I just need a few billion, to build a server farm, to build the machine god, who will tell us how to solve the homelessness and housing insecurity." If it works? That'd be neat. Right now, it sounds like crackhead logic.

Today I’ve compiled a few thousand classes of Javadocs in .978 second. I was so impressed, with a build over 2 minutes, each byte of code we write takes a second to execute, computing is actually lightening fast, just now when it’s awfully written.

Time of executing bytecode << REST APIs << launching a full JVM for each file you want to compile << launching an LLM to call an API (each << is above x10).

The point is that you call the LLM to generate the code that lets you talk to the API, rather than writing that glue code yourself. Not that you call the LLM to talk to that API every time.

Exactly.

As if the challenges in writing software are how to hook APIs together.

I get that in the webdev space, that is true to a much larger degree than has been true in the past. But it's still not really the central problem there, and is almost peripheral when it comes to desktop/native/embedded.

> LLMs offer the single biggest advance in interoperability I've ever seen.

> ... we need to build APIs with just enough documentation (and/or one or two illustrative examples) that an LLM can help spit out the glue code needed to hook them together.

If a developer relies on client code generated by an LLM to use an API, how would they know if what was generated is a proper use of said API? Also, what about when lesser used API functionality should be used instead of more often used ones for a given use-case?

If the answer is "unit/integration tests certify the production code", then how would those be made if the developer is reliant upon LLM for code generation? By having an LLM generate the test suite?

And if the answer is "developers need to write tests themselves to verify the LLM generated code", then that implies the developer understands what correct and incorrect API usage is beforehand.

Which begs the question; why bother using an LLM to "spit out the glue code" other than as a way to save some keystrokes which have to be understood anyway?

The developer has the LLM write the test suite, then the developer reviews those tests.

This pattern works really well.

"other than as a way to save some keystrokes which have to be understood anyway?"

It's exactly that. You can save so many keystrokes this way.

a test suite is a way to stochastically reduce certain classes of risk

it doesn't assert, or imply, correctness in the sense that is used here

Agree. I often prefer to screen scrape even when an API is available because the API might contain limited data or other restrictions (e.g. authentication) that web pages do not. If you don't depend on an API, you'll never be reliant on an API.