Yep. I wonder why there is not more features that would differentiate your LLM API, like for example the Functions in OpenAI LLM APIs. While not perfect it is extremely useful and not aware of a similar offering (I do know that there are Python packages offering similar functionality but it is my understanding that they don't work as well as the OpenAI Functions).

Aye, open models from Meta are tearing at the moat.

> The winner is going to be the consumers

Cloud infra may be a comparable market, since computation is a big share of AI costs. Did consumers win big from competition between AWS, Azure, and GCP? Not sure. I see an uptick in write ups saying “We switched off cloud and reduced costs by 2/3rds.” Not a scientific sample but may leave the question open.

Can confirm: The cloud computing fad is well underway to dying...that is why AI is booming. One need only to follow the wall street dollars to figure that one out.

Several very big profile names have recently begun moving back to self-hosted, hybrid, or dedicated hosting solutions.

Cloud computing never was good in terms of value, however, it was only good in terms of scalability. AI solutions built on top of 'the cloud' will always be even worse.

Note that I pay a certain "third tier" cloud provider less than $100/mo total for hosting large websites that would cost me more than 10 grand a month on AWS/Azure/Google...while having better uptime. (the biggest differences? the complete lack of IO and bandwidth charges, and much lower storage charges)

That should tell you all you need about these types of bubbles, but then again, most of us that watched the entire tech field unfold since the pre-internet phase already knew this.

> Several very big profile names have recently begun moving back to self-hosted, hybrid, or dedicated hosting solutions.

Could you provide some examples?

Dropbox has been the biggest name who did the whole “cloud repatriation” thing, which was all the rage at the beginning of 2023, with claims that the cloud was soon to be dead—but cloud revenues are supposedly expected to be in excess of $1T by 2026, so whatever.

Some random survey from ESG found 60% of respondents repatriated at least some workloads. Who knows what the N was though.

Source: https://www.sdxcentral.com/articles/thinking-about-cloud-rep...

Frankly, it makes sense that companies are deciding what works best and where. But the death knell of cloud providers is simply not happening.

Which cloud provider is it, if you don’t mind sharing?

Hetzner or Oracle, maybe?

Just gonna chip in here and say that anything costing a 100 bucks is not valid as an argument in a conversation about cloud.

The prime selling point for cloud for large enterprises was (and still is):

- a signatory that shares blame on several core security issues (iso stuff) - high amount of flexibility for individual teams used to asking for a vm then waiting four weeks for the itops dept to bring it online

Now the vast majority of cloud moves for large enterprises ends up as a shitshow due to poor implementation sure, but the key points for getting it sold are still there.

And ofc you CAN still cost optimize with cloud, its just harder.

Context: Worked in post and pre sales over some years in MSFT in the enterprise segment.

I got out before the downturn and everyone talking about cost, but my approach in selling azure to the c-suite would be fairly similar today I reckon.

There are a HUGE number of startups that would have had a massively harder time if they had to roll their own infra. Who cares if the companies eventually have to move off? (Though even Netflix seems ok with it overall for now). There are a ton of services that just wouldn't exist otherwise

I disagree, at early startup scale you don't need much, you just buy a better VPS when you need to scale up

Learning aws vs learning how to operate a vps are of comparable complexity

It took me less then a day to setup infra for my startup (more than 10 years ago)

If you're just running VPS on the big clouds, you're not going to get much advantage out of it, indeed.

But tell me what third tier cheap provider has managed scale-to-zero-or-infinity functions? Managed storage with S3-like API? Where can I get an API gateway cheaper than Amazon? What about managed databases? These tools allow me to develop insanely scalable software incredibly easily.

Agreed - all the big clouds are very expensive VPS hostings. Don't use it for that.

You get portability. Which the functions do not provide. Open source solutions have a longer career utility than proprietary offerings. I remember when NetWare certs were all the rage. Useless now. I remember msce. But if you learned open tools 35 years ago instead... You get the picture.

You can scale from 1 thread 512MB RAM, to 500 threads and 12TB of RAM (off the shelf). Which is good enough for almost everyone who isn't planet scale.

Auto scaling also comes with auto billing. Oops, your accidental infinite loop spawning functions has bankrupted your company. You don't have that risk starting with a VPS.

I agree completely, but the argument remains the same - there's not much utility in using the big clouds as VPS providers, and it's definitely costly to do so.

That part of true. Using EC2 at AWS now and it's much more expensive than vultr, Hetzner, or akamai cloud.

> Managed storage with S3-like API?

What's third tier here? Pretty much all of them offer S3 compatibility it's basically table stakes

Sorry not sure if I'm missing something obvious but isn't a VPS gonna be hosted by a cloud provider? How would that be an alternative to using the cloud?

It's not that hard. Particularly for simple deployments, which a startup should have or they are doing it wrong.

How would you do it for a simple webapp? Genuine question

Ingress that handles SSL. nginx, or caddy. Then stand up your app server behind that on the same VM. Database can be on the same or different VM.

I try to not use anything else if I can avoid it on a new project.

Ingress gives you the ability to load balance and is threaded and will scale with network transfer and cpu. Database should scale with a bump in VM specs as well, CPU and disk IOPS.

If you keep your app server stateless you can simply multiply it for the number of copies you need for your load.

Systemd can keep your app server running, it you docker it up and use that

> If you keep your app server stateless you can simply multiply it for the number of copies you need for your load.

This right here so many times over.

I'm not going to spend time worrying about scaling, I'm going spend time figuring out how to make stuff stateless.

Shove your state into the database. Done.

No that's all fine, I mean physically, where would I put a server and stuff if we didn't have cloud providers? I'd need to pay an isp for an IP address and maybe port forward and stuff like that right? I don't get why I wouldn't just do what you mentioned on a five dollar digital ocean droplet or an ec2 instance or whatever, the cloud still seems orders of magnitude easier to get off the ground.

If you rent a cloud vps as an ingress you can run an overlay network and your actual hosted services can literally be anywhere. See nebula, netbird, etc. You can also ssh forward, but that doesn't scale well past a handful of services and is a bit fragile.

For new small systems I suggest you start with a cloud VPS. If traffic is low, cost of downtime is low, and system requirements are high then a cheap mini PC ($150) at the home or office can keep your bill microscopic. If your app server and database are small then you can just throw them on the VPS too.

I run light traffic stuff at home in a closet so it doesn't occupy more costly cloud RAM. Production ready saas offerings I'm trying to sell right now are all in the cloud. Hosting all my stuff in the cloud would cost me hundreds per month. My home SLA is fine for the extras. I don't need colocation at this time, but I have spoken with data centers to understand my upgrade path.

You can run a live backup server at a second location and have pretty good redundancy should the primary lose power or connectivity.

When system requirements elevate (SLA, security, etc) you probably want to move into a data center for better physical security, reliable power and network. Bigger VPS is fine if it is big enough. Can also do a colocation if you don't want to rent, and you contract directly with a data center. I wouldn't look at colocation until your actual hosting needs exceed at least $100/mo and you're ready for a year long commitment.

But to the point of the original question that started this thread - the takeaway is still that cloud services made development massively easier right? The answer to the original question seems to still be "Yes cloud providers did lead to big wins for customers" since none of these other suggestions are able to get away from needing a cloud service provider without making starting something intensely difficult. And you wouldn't be able to get a vps for 5 bucks for ingress without all the other cloud competition in the market.

> Cloud infra may be a comparable market, since computation is a big share of AI costs. Did consumers win big from competition between AWS, Azure, and GCP? Not sure. I see an uptick in write ups saying “We switched off cloud and reduced costs by 2/3rds.” Not a scientific sample but may leave the question open.

Renting a server and using standard commodity open source software and standard (outsourced) sysadmins is way cheaper and faster than learning and dealing with all the proprietary AWS and Azure junk.

(Probably also less reliable than "cloud", but who cares if you're a startup.)

Much harder to switch cloud providers than to switch LLM models. How much time would it take most companies to move their product from AWS to GCP, for example? What if you use a cloud specific tool like DynamoDB?

Margin is a function of stickiness/cost of switching (among other things).

I suspect eventually we will enter a world where migrating cloud providers is mostly a click of a button, but we're a long ways off from that. Requires vendor agnostic and portable apis/containers/WASM runtimes everywhere.

Swapping an LLM, at least in the current state, is about as close to updating a pointer as you can get

Well I'd have to change my Terraform provider and the managed Kubernetes resource... Other than that, it'd be the same. So half an hour of coding + half an hour of reconfiguring CI secrets?

Pretty obvious to most that switching cloud providers is not quick or painless for the majority of orgs. There's not really an argument in good faith to suggest otherwise.

Especially given that many orgs use managed or cloud specific solutions that have no 1:1 mapping between vendors

I said I'd have to change that one managed resource. Actually it's two - the managed database. But no more.

> I said I'd have to change that one managed resource. Actually it's two - the managed database. But no more.

As someone who worked on a similar project recently, I'm getting that idea that you obviously don't know what you're talking about.

Didn't you know? It's only one DB ENV variable and you're swapped over from GCP to AWS. /s

I'm running the tech for a global startup with 100M EUR turnover. It's still pretty small, but it's something.

You need to plan ahead, sure. Portability was one of my main concerns (including the possibility to go self-hosted). But it's definitely not impossible, nor too hard to do.

You lost me at Terraform and Kubernetes..

What's a serious cloud provider-portable replacement for Terraform?

And Kubernetes? It goes way beyond containers, you can replace many cloud provider-specific resources with Kubernetes resources. What alternative gives you that?

If you swap an LLM, at the very least, you have to run your entire Eval set again.

This will almost certainly lead to a prompt rebuild, to better accommodate new model idiosyncrasies.

If you are unlucky- your use case may be one where Evals require human review.

Unless you are YOLOing it without evals. In that case this is relevant.

You need to rerun eval after each LLM update. GPTs have a new version every couple of months and their capabilities can change quite drastically pretty much randomly. Maybe they will make it more robust in time, but I think this is the feature of the technology and people will have to adapt to these quirks

Yup. Constant Eval.

It's not the tech it's the data. As far as I know the data is not freely shared or at the very least there will be custom built models from data you can't get anywhere else.

But you also have to account for the counterfactuals.

If the cloud had not existed, those that claim they saved money switching away from cloud might never have been in business in the first place.

Certainly the undisputed winners will be the very few firms with enough engineering resources and GPUs to train their own models (not just fine-tune) where the models in question increase the productivity of workers in their non-ai-related profit centers. After that we have the real question of what the future will be of open source LLMs, on the one hand, and the question most relevant to this article of what sort and whether profitable “AI businesses” can be sustained over time. As Stratechery has analyzed, it is very possible that OpenAI turns out to be a very profitable B2C company with ad revenue in ChatGPT not concerned with their B2B sales or even the objective quality of their AI. Right now is an incredible time for AI: cheap Uber rides never qualitatively changed my life, but the current consumer access to AI models is truly incredible and I hope that only improves. However, even ignoring whatever happens on the regulatory front, I don’t think that is guaranteed at all.

This was definitely a theory that made people burn tons of money on the past couple of years, but I don’t think it holds water. These models are getting obsolete so fast, and there’s so many open ones, I doubt any one’s privately trained model can stay relevant for long

The data is the moat.

(If you can train your internally deployed LLM on data none of your competitors have, that's an advantage).

In that case, X.AI, powered by X/Twitter/Tesla data and possibly Facebook (both closed, and somewhat hard to crawl inside) have the largest moat.

I don’t think they necessarily will be allowed to train on their data unless they get explicit permission. They will try, but the way I see privacy revaluations is that users will have to authorize specific uses of their data and not be surprised by any application.

This could be one of the more interesting privacy fights of the next decade.

I’m sure there are easy cynical takes about how they will just shrink wrap the EULA, and maybe they will. But in a good privacy environment, users should never be surprised and have control over how their data is used. And I think we’ve made some progress there.

> I don’t think they necessarily will be allowed to train on their data unless they get explicit permission. They will try, but the way I see privacy revaluations is that users will have to authorize specific uses of their data and not be surprised by any application.

If there's one company that I don't think cares about user permissions or the law, it'd be Twitter.

The EU officially warned Elon about DSA fines and the response was less than serious.

https://www.cnn.com/2023/10/10/tech/x-europe-israel-misinfor...

China probably has the most comprehensive data on its users from a surveillance perspective

You may not need to train a model to make use of your data though. Maybe a cheap fine tune would work just as well. Maybe just having the data well indexed and/or part of the prompt context is good enough.

Idk. These were trained on pretty public things like Wikipedia.

It's not anymore. If the model is publicly accessible, its skills can be distilled by performing some API calls and recording input-output pairs. This scheme works so well it has become the main mode to prepare data for small models. Model skills leak.

I agree, publicly deployed models seem to be easy to train from. I did say "internally deployed LLM" though. agentcoops said "...where the models in question increase the productivity of workers in their non-ai-related profit centers" above, that's the bit I was thinking about. I think private models, either trained from scratch or fine-tuned, are going to be a big deal though they won't make the PR splash that public models make.

The conclusion for that seems to be that it just yields a model that has the surface look and feel of GPT3 or 4 but without the depth, so the experience quickly becomes unsatisfactory once you go out of the fine tuning dataset.

It seems to be compelling because a genie has been the promise of technological progress forever and let's be honest, it's been marketed as such. Why would people not invest in that unless you were a technological savvy skeptic like yourself.

Because the cost of entry to the market is so absurdly high right now, it is seen as a good investment opportunity. If you throw enough money at it you can make your place no nice and early and then win later by sheer experience in the field. That is the idea.