This is entirely unsurprising. A language generator is not designed for analytical analysis. The fact it codes well is what I think is throwing people. Since code is a form of language it works well for the language model. The issue with code will come from larger context where analytical thought is needed and it will break down.

People are trying really hard to make "search" or something like it the killer app for LLMs but from what I've seen, it's translation. Coding in its purest form is a kind of translation, from natural language requirements to more precise computer languages and vice versa. Keeping an LLM properly focused on that task, it's pretty astonishing what it can do. But as you say, assuming it can do anything more than that is a trap.

> Coding in its purest form is a kind of translation, from natural language requirements to more precise computer languages

Unfortunately not so true. A simple way to check this is to look at what the brain is doing. Coding barely triggers the language components of the brain (your usual suspects like the Vernicke's and Broca's areas + other miscelleneous things). In fact what you'll find is that coding triggers the IP1, IP2 and AVI pathways. These are part of the multiple demand network. We call them programming "languages" because we can represent them in a token-based game we call "language" but the act of programming is far from mere language manipulation.

> from natural language requirements to more precise computer languages

This makes sense. Translation to precise code would include more of the logic/planning bits of the brain, since that's the context of the output. I suspect a person fluent in Lojban[1] would show similar areas activating during translation from English to Lojban. I also assume translating code to words, with documentation, would include more language, since that's the context of the result.

That doesn't mean it's not translation. It's just translation to a precise description of intent.

1. https://en.wikipedia.org/wiki/Lojban

> I think these kinds of studies are of limited use. I don't believe raw GPT4 is that helpful in the enterprise.

I'm not so sure about this. A lot of work in and for especially large organizations is writing text that often doesn't even get read, or is at most just glanced at. LLMs are great at writing useless text.

That kind of text is valuable as a signaling exercise - the author is willing to claim the text as indicating their actual beliefs. Writing it with AI would actually damage the author's credibility since it's no longer their words - although this shouldn't be a problem if the author is carefully checking the text every single time to make sure it represents their position.

The effect would be even worse the first time the author missed a mistake from the AI and had to walk it back publicly. Nobody's going to trust anything they put in writing for a while after that.

There are genres of text that have little to do with the writer's beliefs and are a kind of formality with contents known by all interested parties beforehand. Sort of elaborate prose versions of "I Agree With the Terms and Conditions" textboxes.

At first I thought this was a flippant joke, but I can actually see the value in saving time writing junk without having to alter existing bureaucratic processes from which nobody gets much value.

I am using ChatGPT to write the genetic boilerplate that e.g. many funding applications require. Ethics statements or inclusivity policies etc where the contents are largely predefined and known by everybody but still have to be written in prose instead of e.g. clicking boxes.

Isn't most of science like that too?

There is a lot of boilerplate and fluff in scientific papers that everybody knows is fluff.

> GPT4 alone will especially not be that useful for analytical work. However, developers can make it a semi-capable analyst for some use cases by connecting it to data lakes, describing schemas, and giving it tools to do analysis. Usually this is not a generalist solution, and needs to be built for each company's application.

I've been thinking about this a lot lately. I'm a data analyst and all these "give GPT you data warehouse schema, let it generate SQL to answer user's queries" products completely miss the point. An analyst has value as a curator of organizational knowledge, not translator from "business" to SQL. Things like knowing that when a product manager asks us for revenue/gmv, we exclude canceled orders, but include purchases made with bonus currency or promo codes.

Things like this are not documented and are decided during meetings and in Slack chats. So my idea is that in order to make LLMs truly useful, we'll create "hybrid" programming languages that are half-written by humans, and the part that's written by LLMs when translating from "human" language is simple enough for LLMs to do it reliably. I even made some weekend-prototypes with pretty interesting results.

The first generation of company-focused llm will need to be bespoke. Teams involved with this will learn the necessary abstractions and will build new companies to lower the on-ramp or join the LLM providers to do the same.