If we get more unexpected emergent abilities by scaling the model further, things could get very interesting indeed.

I have no idea if there are or there aren’t, but that’s the big question.

FWIW, I do find that Claude (Anthropic's GPT) is often better than GPT4 -- and very fast. Entrants can compete on price, safety, quality, etc.

Wouldn't be surprised at all if the major API-based vendors start leaning in on making their safety config proprietary.

If a business has already sunk XXXX hours into ensuring a model meets their safety criteria for public-facing use, they'd rather upgrade to a newer model from the same vendor that guarantees portability of that, versus having to reinvest and recertify.

Ergo, the AI PaaS that dominate at the beginning will likely continue to dominate.

Fine tuning is at a low point now, but i expect this to create a moat for the same reasons.

It also provides a ChatGPT interface, and a number of other models.

Prompt:

  The original titles and release years in the Harry Potter series are:

  Philosopher's Stone (1997)
  Chamber of Secrets (1998)
  Prisoner of Azkaban (1999)
  Goblet of Fire (2000)
  Order of the Phoenix (2003)
  Half-Blood Prince (2005)
  Deathly Hallows (2007)
  
  Given this, generate a new Harry Potter title, using only the words found in the existing titles. Avoid orderings in the original titles. You may add or remove plurals and possessives.

ChatGPT: Blood Chamber of the Phoenix's Prisoner

Claude-instant: Chamber Prince Half-Blood Phoenix

So when I’m doing quantum computing work, I go back and forth between Claude and GPT4 and both complement the other very well.

[0] https://en.wikipedia.org/wiki/Pascal%27s_wager

O̶p̶e̶n̶AI.com will eventually have to raise their prices which is bad news for businesses not making enough money and still are sitting on their APIs as O̶p̶e̶n̶AI.com themselves are running up huge costs for their AI models in the cloud for inferencing.

Anthropic is just waiting to be acquired by big tech and the consolidation games will start again.

If you're quabbling over how much OpenAI charges for an API today that barely just launched and from which we have barely scraped the surface for applications... I don't know that seems like a failure to think broadly and assumes the market today is what it will look like in 5yrs.

There could be a ton of lucrative businesses which subsidize those operating costs. It doesn't have to be a mega-company like Google that floats it indefinitely off their ad business, or whatever other scheme. We have no idea what the value of those APIs are or if the API is the real business they (and others) are going to be relying on in the long term.

Do you have data supporting this or is it just speculation? Given we don't even know how many parameters GPT-3.5 and GPT-4 have, yet alone how efficiently they are implemented, I don't see how we can go about coming up with an accurate estimate for the cost per token.

R&D heavy markets might have some different characteristics but it's still way too early to say with AI.

It's not the software or hardware that will "win" the race, it's who delivers the packaged end user capability (or centralizes and grabs most of the value along the chain).

And end user capability is comprised of hardware + software + connectivity + standardized APIs for building software on top + integration into existing systems.

If I were Nvidia, I'd be smiling. They've been here before.

My dad told me a quip once: "It's amazing how much luckier well prepared people are."

But i think it is underestimated how important it is for the model to be uncensored. ChatGPT is currently not very useful beyond making fluffy posts. As a public model, they won't be able to sell it for e.g. medical applications because it will have to be perfect to pass regulators. It cannot give finance advice. Censorship for once is proving to be a liability for a tech company.

In-house models OTOH can already do that, and they can be retrained with additional corpus or whatever. And it's not even like they require very expensive hardware.

In the GP's scenario, I wouldn't be building either piece of software.

…I mean, “not-bad-at-all” depends on your context. For doing mean real work (ie. not porn or spam) these tiny models suck.

Yup, even the refined ones with the “good training data”. They’re toys. Llama is a toy. The 7B model, specifically.

…and even if it weren’t, these companies can just take any open source model and host it on their APIs. You’ll notice that isn’t happening. That’s because most of the open models are orders of magnitude less useful than the closed source ones.

So, what do want, as an investor?

To be part of some gimp-like open source AI? Or spend millions and bet you can sell it B2B for crazy license fees?

…because, I’m telling you right now; these open source models, do not cut it for B2B use cases, even if you ignore the license issues.

Eventually there will be a good enough model for most personal uses, our personal AI OS. When that happens there is a big chance advertising is going to be in a rough spot - personal agents can filter out anything from ads to spam and malware. Google better find another revenue source soon.

But OpenAI and other high-end LLM providers have a problem - the better these open source models become, the more market they cut underneath them. Everything open source models can do becomes "free". The best example is Dall-E vs Stable Diffusion. By the next year they will only be able to sell GPT4 and 5. AI will become a commodity soon, OpenAI won't be able to gate-keep for too long. Prices will hit rock bottom.

I really don't think you understand just how absurdly high the cost is to train models of this size (which we still don't know for sure anyways). I struggle to see what entity could afford to do this and release it as no cost. That doesn't even touch on the fact that even with unlimited money, OpenAI is still quite far ahead.

You can also run your stack on a single VPS instead of cloud, gimp instead of photoshop, open street maps instead of Google maps, etc.

There will always be companies who can benefit from a technology, but want it as a service. In addition, there will be a lot fine-tuning of LLMs for the the specific use case. It looks like OpenAI is focusing a lot on incorporating feedback into their product. That’s something you won’t get with open-source models.

The application of AI to business problems will be lucrative, but the models are just a tool and the money will come from the domain-specific data (i.e. user and business data), which Microsoft, Google, and even Meta are positioned for. Having a slightly better model but no customer data or domain expertise doesn’t seem like a great recipe.

Then again it’s AI, so there’s more uncertainty than the commodity market. Maybe Anthropic will surprise and I’ll be as wrong about this as I was about OS/2 being the future. But I’m very skeptical.

Think of LLMs as the understanding component in the brain, once you can understand instructions and what actions need to happen from those instruction you’re done.

The rest is integrations, the arms legs and eyes of langchain. Then memory and knowledge from semantic search, vector databases and input token limits.

The LLM is but the core of the entire ecosystem. Just like how MLOps is 99% of the work, choosing an LLM is 1% of the effort in the final product.

The correct term for this is “pyramid scheme”.

But if early investors only profit due to late investors pouring money in, that’s by definition a pyramid scheme.

To say a market that large will be owned by only 4-5 companies doesn't make sense. Let's take the PC market for example: there are roughly 6 companies that make up ~80% of the market, sure. However, let's look at a tiny participant compared to the total market (~65B): iBuyPower at rank #77 had sales of 40MM or 0.06% (small, expected) of the market with a much smaller capital investment. If look at this percent compared to 5T, we would be at 3B. While the 5B investment stated in the headline could result in a lower ranking and smaller share, the point stands that there is still a lot of money to be made on the long tail. Even if Anthropic fails, there will be other companies with similar infusions that succeed.

Goldman Sachs Research just pushlished their own analysis as well. [2] Their conclusions are "As tools using advances in natural language processing work their way into businesses and society, they could drive a 7% (or almost $7 trillion) increase in global GDP and lift productivity growth by 1.5 percentage points over a 10-year period." and "Analyzing databases detailing the task content of over 900 occupations, our economists estimate that roughly two-thirds of U.S. occupations are exposed to some degree of automation by AI. They further estimate that, of those occupations that are exposed, roughly a quarter to as much as half of their workload could be replaced."

[1] https://arxiv.org/pdf/2303.10130.pdf

[2] https://www.goldmansachs.com/insights/pages/generative-ai-co...

From [2]: "Analyzing databases detailing the task content of over 900 occupations, our economists estimate that roughly two-thirds of U.S. occupations are exposed to some degree of automation by AI."

These are people who do not understand the jobs they are claiming AI will do. Ultimately, I think they are not doing much better than guessing.

- Alan Kay

At this point in time, it’s hard to tell whether moats will arise around large language models. Peter Tiels thinks so or he wouldn’t have invested (see his Competition is For Losers presentation).

What is unlikely is that semi-good companies will thrive. Maybe for a few years but at some point the smaller players will be pushed out of the market or need to find a specific niche. Just look at cars to see this. Around 1900 there were hundreds of car brands.

> let's take McKinseys rough estimates of job displacement (~30% of ~60% of jobs, ~20% of work)

I think we forget that our perspective of AI now is comparative, probably to that of a preindustrial worker worried about machines. Displacement, sure but complete replacement seems a non nuanced view of how it may all turn out.

I mean, maybe they won't like you say, but what if they do? Then you're probably screwed. Better to gamble a few billion, imho.

This is a ridiculously myopic statement. Foundation models are an extremely powerful technological advancement and they will shake the global economy as very few things did in human history. It's hard to imagine how this is not obvious to anyone right now, specially here in this forum.

It's a high risk investment at this stage but the money is being thrown at the people as much as the current business plan.

The question is whether whomever builds them can make a profit doing so, or will they just end up being the suckers that everyone who actually makes money piggybacks off. It's really not clear at the moment.

The game theory logic doesn't care about the labels "OpenAI" or "Anthropic" or any of the others, it's the same if you switch it around arbitrarily, but this is easier to write about if I focus on one of them:

At some point, someone will reproduce GPT-3.5 and ChatGPT, given how much is known about them. When that happens, OpenAI can't make any significant profit from it. GPT-4 might remain sufficiently secret to avoid that, but the history of tech leaks and hacks suggests it too will become public, but even if it does itself remain behind closed doors, there is a further example in that DALL•E 2 is now the boring 3rd horse in the race between Stable Diffusion and Midjourney, and the same may happen with the GPT-series of LLMs.

The models leaking or being superseded by others implies profit going to increased productivity in the general economy without investors getting a share.

It also sounds like you believe you have defined the bounds for what AI will be, and figure we'll just iterate on that until it's a commodity. I don't think AI will be that static. We're all focused on stable diffusion and LLMs right now but the next thing will be something else, and something else after that. As each new technique comes out(assuming they are all published), we'll see quick progress to incorporate the new ideas into various implementations, but then we'll hit another wall, and suddenly big budgets and research teams may matter again.

tldr is that it is way too early to make the cynical claim you are making.