“What that also means is that you don’t need to be an AI genius to succeed in building applications. With thoughtful software engineering and a focus on customer data, you’ll build a moat over time.”

This sounds encouraging at first glance, but even more demoralizing when you think about it. It doesn’t matter how clever or smart you are over your competitors. If you don’t have the data, you don’t stand a chance. And of course the incumbents have the data not you, the entrepreneur

I don't see it that way. If you build a genuinely novel application then the critical data doesn't exist yet (IMO almost by definition). Sometimes it's easier for a startup to do this rather than for an incumbent who's trying to shoehorn the application into a preexisting framework (technical, operational, whatever).

If the data doesn't exist, you can't train an (neural network) AI as they exist today.

Whatever thing people build, will be based on data that exists, not data that will be created afterwards. And guess what, you don't have that data.

You could bootstrap a niche model using what's available and go from there. I don't think it's all that different from any other kind of bootstrapping startups tend to require.

Bootstrap a niche model based on what?

On the yet to exist data?

> The real differentiation lies in your data you feed into your models.

It's more than data. Steven Jobs used to tell the founders of the Segway that their secret technologies would leak sooner or later, even if they set up their factories in the middle of a desert in Nevada. His advice to the founders was that they needed to build a product that users couldn't get away from even if all the competitors had the same technology (or so I remember). That could be a process, an ecosystem, market penetration with an amazing supply chain (think about the largest seller of straws in the world. The unit price of a single straw is close to 0, which is really hard to achieve), and etc.

To me, data will just be one key component of the moat but not all. The moat of AI is the same ol' entire ecosystem: an infrastructure that is so efficient that the company can keep driving down the unit price of hosting the AI models; a fabulous culture to enable the company to keep churning out improvements; an extensive data platform and the associated process and sources that keeps provisioning quality data, a number of killer applications...

> We firmly believe the moat for AI application is in the data and the data engineering today. At some point, the process of building custom LLMs might get so fast and easy that we’ll all return to building our own models. That simply isn’t the case today.

Customized small models will outperform larger general models for your specific use case.

No they wont. Any model you train now will be beat by GPT5 easily

Past performance is not a predictor of future performance.

While it's possible the gap between GPT5 and 4 is as big as between 4 and 3, it's unlikely. The gap between 2 and 3 was much larger as the one between 3 and 4 (and similarly between 1 and 2).

Also, it's not clear that GPT5 will do this in an *economical way* once the spigot of investor money stops.

I think the real power of customized small models will be running things on local hardware, except that we're in an awkward phase where the local hardware isn't quite beefy enough to run anything really useful yet. Maybe Apple will do something interesting in that space at WWDC.

Also not feasible. A network request to groq type machines will outperform your local hardware by such a huge amount that it wont make sense other than some very niche tasks

Network availability, latency, privacy, etc. many qualities to consider beyond model size and performance for applications.

And cost-efficiency, if I'm using an LLM as an Siri-like assistant on my phone, most of the tasks I'll want it to do won't be that complicated and it would be a waste to send them to some SOTA LLM in the cloud, which I'll have to pay for by a monthly subscription or on a per-token basis.

Except nobody but groq has such type of machines, and the economics of cloud AI is very hard to make it works in practice. Offloading the capital cost (which is the hardest kind of cost to swallow for a company) to customers is very compelling business-wise.

What they are doing is not very special soon a lot of companies will do it. Lets call it end to end LLM hardware

And they will be in everyone's laptop, because that's pretty much the only economically viable way way to amortize the development cost ;)

Not for most human language, or anything that requires business-specific context where what's publicly available lags behind the state of the art your business cares about.

And of course, not if you care about token throughput more than fancy abilities. Or price for that matter.

So for many if not most businesses needs GPT-4 isn't the best tool out there, and GPT-5 is the canonical example of a vaporware right now.

Tell more more about the data moat.

I did expect that too, but it isn't happening reliably.

But small customized models seem to perform close to as well as large general ones.

If you can't build a moat with technology you build it politically which Altman is already trying to do.

Before I clicked I expected the article to be about LLM's failures in structural engineering applications

This is probably why Google or Meta will "win" over OpenAI in the end.

They have all the data, and no one else even comes close.

Microsoft has plenty of data too. In Microsoft Teams, LinkedIn posts and messages, and Outlook emails.

I wouldn't worry about microsoft delivering quality in anyway.

Plus every company's files in OneDrive and SharePoint.

Don't forget they have github as well.

Microsoft 365 (nee Office 365) as well. And Dynamics 365. And GitHub. And OneDrive. And SharePoint. And Power Platform.

Honestly I think they might have more useful data than Google, given Bing knows more or less that same as GoogleBot. Meta doesn't come close, unless you want your LLM to be purely conversational.

Nobody has explained how they could use that data without producing a model that would emit private information.

Perhaps de-identification before training could be helpful here.

Microsoft does seem active in this, e.g. https://microsoft.github.io/presidio/

None of that stuff actually works. You can remove someone's social security number from the data but there is still only one person at the exact intersection of all the data that isn't individually considered personally identifying but collectively it still is.

Moreover, that isn't even the problem here. Suppose your company has a trade secret. You know how to manufacture widgets more efficiently than your competitors. If Microsoft produces a model that will now tell your competitors your secret process that it learned from your internal emails, it's completely irrelevant whether they stripped the PII out of your emails first.

And if they use any of it the entire worlds corporate lawyers will show up on their doorstep

Unlike googles victims (individuals) corporations can and do fight back when someone plays it fast & loose with their confidential coms

Large companies like Google will fail at making the most successful LLMs because of internal cultural problems.

Go away ChatGPT

So annoying. I feel sorry for people who actually read that paragraph.

...which is why OpenAI is focusing more on enterprise sales and unique value-propositions such as the GPT Store which can't trivially be imitated by competitors.

This post seems to misunderstand what a "moat" is in a business sense (and unfortunately a lot of AI hypesters on social media do as well). The fact that LLMs are becoming a commodity was the point of the original "OpenAI has no moat" memo by Google, which has proven to be accurate.

"It might feel like your applications’ prompts or prompt templates are a good form of differentiation. After all, your top-notch engineering team has invested days into tuning them to have the right response characteristics, tone, and output style. Of course, giving your competitors your prompts would probably accelerate their progress, but any good engineering team will figure out the right changes quickly. The main reason is that the experimentation (with the right evaluation data!) is quick and easy — trying a new prompt template isn’t much harder than writing it out. All it really takes is a little bit of patience, some creativity, and extra Azure OpenAI credits."

And yet, over and over, I see products with output that clearly comes from simple and frankly lazy prompting. You can do a lot with prompting, but engineerings are not putting in the work! (If an engineer is even the right person... probably not, given any specific application of an LLM.)

Prompting also isn't so reductive that you just write an evaluation and then iterate on the prompt until you satisfy the evaluation. Prompting is a co-creative exercise between the LLM, the domain expert, the product, and the user. And sure "data" fits in there, as well as relationships, comprehensibility, workflows, etc etc... the AI component is just a small piece of any full application.

the best model is not GPT-4 it's April of 2024

What's the best model now?

Claude 3 Opus

Opus is an order of magnitude more expensive and slower than GPT-4.

doesn't matter that it's more expensive though, it beats it in benchmarks

also, it's not "an order of magnitude" more expensive

There’s nothing you can do to make AI apps reliable. You can experiment all you like and play with prompts… in the end you will have a flakey product.

Hallucinations are features not bugs and all that.

You can build a moat and fill it with bullshit scraped from Reddit. That's all AI is anyway. Sorry to burst anyone's hype bubble.

“ For reference, our team at RunLLM has spent roughly 70% of our engineering cycles in the last quarter on data engineering — everything from pre-processing data at ingestion time to implementing hybrid search to reranking results.”

This to me is why I am actually bullish on building ‘LLM wrappers’ that are sustainable businesses. While you might not be developing new technology, you can be adding value to a client who can use chat gpt and “talk to your docs” but not the surrounding work to make it frictionless for their workflow or company and actually build a business. It’s entirely possible that AI will get so good that it makes it easy to all the surrounding tasks too.