News for hackers. Stuff that matters.
Re:Mistral AI raises $640M at $6B valuation by t-writescode on Tuesday June 11, 2024 @05:00PM (#40651406)
All this external money raising when it comes to AI scares me.

For companies that are built off these AI services, there's a very real fear that the prices we're getting on API calls aren't covering the costs of the requests themselves under some assumption that costs will magically go down, or we'll get locked in.

I hope the companies like Mistral and OpenAI know that if they're selling API access below cost they could be the origin of a __lot__ of companies closing and choose to make sure their operating costs are sustainable. (I understand creating new features and new models taxes additional capitol and that's a perfectly fine use of VC money. But selling a $1.00 for $0.90 is dangerous and bad.)

Re:Mistral AI raises $640M at $6B valuation by d_watt on Tuesday June 11, 2024 @05:08PM (#40651505)
It's due diligence to understand your cost of goods sold using open source models where possible.

Don't actually spend the business effort to use them if it's not worth it, but have a business continuity plan to be able to switch over, especially if a "single source provider" going out of business or changing it's pricing could represent a existential thread to your business.

It's the same as any other tech vendor.

Re:Mistral AI raises $640M at $6B valuation by popalchemist on Tuesday June 11, 2024 @05:37PM (#40651854)
API prices of the private AI platforms are more or less comparable to self-hosted OSS solutions like Meta's LLAMA 3.

So at least for current gen, you're ok.

Re:Mistral AI raises $640M at $6B valuation by lumost on Tuesday June 11, 2024 @05:23PM (#40651664)
There is an interesting dynamic right now. NVidia is selling cards at somewhere around 98% margin, they are then investing money into these startups - who will then spend some fraction back on NVidia hardware. It's a smart strategy by NVidia, as they can effectively leverage past revenue to create future revenue - but it's unclear what business models will be sustainable coming out of this investment.
Re:Mistral AI raises $640M at $6B valuation by elorant on Tuesday June 11, 2024 @05:37PM (#40651861)
Nvidia’s strategy is to build a cloud service to avoid being locked out once all the other major players develop their own chips. So by owning an LLM they could provide their own service with their own hardware.
Re:Mistral AI raises $640M at $6B valuation by churchill on Tuesday June 11, 2024 @05:35PM (#40651823)
You can only keep an Ouroboros loop going for only so long. At some point, you'll need to inject fresh, demand-driven, customer revenue into the machine.
Re:Mistral AI raises $640M at $6B valuation by yumraj on Tuesday June 11, 2024 @05:35PM (#40651822)
TBH, what should scare you is that most of the AI startups are building on top of someone’s APIs.

So, in most cases they have no moat. They are going to close, and while some will close due to operational cost issues as you identified, a lot more will close since they have no defensible business/revenue model.

Re:Mistral AI raises $640M at $6B valuation by llamaimperative on Tuesday June 11, 2024 @05:38PM (#40651864)
Right, but one group closes by definitionally not doing anything value added. The other group closes by building something value added within a heavily distorted economic environment.
Re:Mistral AI raises $640M at $6B valuation by yumraj on Tuesday June 11, 2024 @05:49PM (#40651995)
Agree. But, if the source API pricing is increased, theoretically any startup that has actually created something of value and stickiness should be able to adapt.

Or, even better if they can price future API price increases in their own pricing, today.

Re:Mistral AI raises $640M at $6B valuation by DevX101 on Tuesday June 11, 2024 @05:37PM (#40651863)
I'd be interested to know the real cost. Are they selling $1.00 for $0.90 or are they selling $1.00 for $.10?

If it's the latter, way more startups on SOTA models will become unviable once the investor subsidies dry up.

Re:Mistral AI raises $640M at $6B valuation by ptero on Tuesday June 11, 2024 @05:07PM (#40651496)
Mistral's open models are not large. If Mistral were to shut down users could easily find another provider to run inference or bring it in house. I think even their commercial models are relatively cheap to run (at least compared to flagships from the big houses), so even if Mistral were to increase the usage prices to reflect true costs it is unlikely to bankrupt most users. My 2c.
Re:Mistral AI raises $640M at $6B valuation by moomoo11 on Tuesday June 11, 2024 @05:18PM (#40651621)
Why’s that bad? If companies were building something that can be trivialized so easily that’s on them.

Imo. A company can use whatever wrapper services but the business they serve should be smarter.

Re:Mistral AI raises $640M at $6B valuation by blackeyeblitzar on Tuesday June 11, 2024 @05:18PM (#40651615)
One thing to consider is that they are EU-based. That means they will meet the EU's regulatory demands (safety, censorship, whatever). They will have a unique place in serving EU customers and look like a safe bet for companies headquartered there. They may also be the default choice for governments in the EU.
Re:Mistral AI raises $640M at $6B valuation by braza on Tuesday June 11, 2024 @05:26PM (#40651708)
Of course, the EU has a large user base and economically strong demographics. Still, even knowing mistral excellence it's hard to conceive that they will serve 27+ countries with different languages, demographics, and cultures.
Re:Mistral AI raises $640M at $6B valuation by nickpsecurity on Tuesday June 11, 2024 @05:04PM (#40651456)
It’s a classic problem. The trick is to make sure you’re not competing on cost alone. Small to midsized companies are better off focusing on other differentiators. The extra money those differentiators brings makes their business more sustainable, too.

You can actually see it in paid, web stacks. There’s free ones, esp FOSS, that are really good. So, the paid contenders try to be better at other things. There’s many of them in the market, too, despite over a decade of free alternatives.

Re:Mistral AI raises $640M at $6B valuation by ToucanLoucan on Tuesday June 11, 2024 @05:01PM (#40651423)
Fear? That's been a fact as far as I've been aware since the big blowup back in mid '23. It got better, ish, with some of the updates they've dropped but it's still catastrophically losing money on every call.

I'm guessing that's going to be the needle that finally pops this bubble, because as neat as a lot of shit going on with various models is, nobody is going to pay what it actually costs to run the things in order to have that neatness.

Re:Mistral AI raises $640M at $6B valuation by tbruckner on Tuesday June 11, 2024 @05:03PM (#40651451)
> catastrophically losing money on every call

You're going to have to bring some actual numbers on that speculation.

Re:Mistral AI raises $640M at $6B valuation by t-writescode on Tuesday June 11, 2024 @05:02PM (#40651433)
Citation? Because I haven't been able to find any such thing in my research.
Re:Mistral AI raises $640M at $6B valuation by ToucanLoucan on Tuesday June 11, 2024 @05:20PM (#40651646)
Some quick googling says OpenAI is losing $700,000 per day operating it's models, which some other quick googling say are processing roughly 10 million queries per day, which if we do quick back-of-napkin math is about 7 cents per query.

Currently their lowest-tier price is $20/month, and limits users to 80 queries per hour. If we assume that 7 cents number and our hypothetical user actually uses those 80 queries for whatever odd reason, that's $5.60 it costs OpenAI, which if that user used them for say, 8 hours (the typical work day), that's $44.80 for OpenAI on a user who's only paid $20. And that's one day, that $20 currently gets them a month of access to that 80 queries per hour. I hope for the dream of profitability that they would get throttled at some point. So the user could, in theory, already render their subscription a net loss for OpenAI with 4 workdays of steady usage.

And yes this is strictly back-of-napkin math here but it would explain why "OpenAI" and "bankrupt" keep appearing in the tech press over and over.

Re:Mistral AI raises $640M at $6B valuation by crazygringo on Tuesday June 11, 2024 @05:08PM (#40651502)
Why does it scare you? Why do you think it's dangerous and bad?

It's a well-established, economically rational business model to subsidize prices in order to gain market share.

Uber has been very successful at it. Amazon has had great success. It's standard.

Investors know what they're doing and know that some companies won't be able to sustain it and will go out of business. It's a calculated, known risk.

And users don't really need to be concerned. Just enjoy the low prices while they last. If the service you use goes out of business, it's pretty easy to switch to another.

Re:Mistral AI raises $640M at $6B valuation by yjftsjthsd-h on Tuesday June 11, 2024 @05:11PM (#40651545)
> Why does it scare you? Why do you think it's dangerous and bad?

> It's a well-established, economically rational business model to subsidize prices in order to gain market share.

Because once they have the market share, they raise prices and/or make the product worse in order to get their money back.

> And users don't really need to be concerned. Just enjoy the low prices. If the service you use goes out of business, it's pretty easy to switch to another.

Well that's the question, isn't it? If the market stays healthy and they don't take the oxygen out of the room and kill the competition and switching costs stay low, then yes by all means enjoy the free ride but plan for its end. But... do plan for it.

Re:Mistral AI raises $640M at $6B valuation by eitally on Tuesday June 11, 2024 @05:37PM (#40651855)
I think the bigger risk is that they don't get the market share and then they're tempted to make unethical decisions to keep afloat as their unbalanced inputs & outputs start to drive increasing pressure from their investors.
Re:Mistral AI raises $640M at $6B valuation by worldsayshi on Tuesday June 11, 2024 @05:23PM (#40651678)
>they raise prices and/or make the product worse in order to get their money back

They should only be able to do that if they have a considerable moat though?

Re:Mistral AI raises $640M at $6B valuation by IanCal on Tuesday June 11, 2024 @05:12PM (#40651553)
> It's a well-established, economically rational business model to subsidize prices in order to gain market share.

Right but it means you can't depend on that price remaining low. If it's below cost, the price must rise. That may be important for anyone building businesses on top.

> it's pretty easy to switch to another.

Well that assumes the others have true lower prices.

Re:Mistral AI raises $640M at $6B valuation by crazygringo on Tuesday June 11, 2024 @05:22PM (#40651657)
> the price must rise. That may be important for anyone building businesses on top.

Well yes but every business owner knows that, or should know that. That's a known business risk. We're all adults here.

If your business fails because you naively assumed that prices you rely on will never go up, then that's on you.

Re:Mistral AI raises $640M at $6B valuation by IanCal on Tuesday June 11, 2024 @05:54PM (#40652045)
> Well yes but every business owner knows that, or should know that.

Know that they are buying services below cost? I don't think so.

It's a reasonable bet that compute costs generally trend downwards, and in a competitive market where the key thing you're paying for is compute that should also trend downwards.

That's different to finding out that everyone's been selling things way under the true cost.

Re:Mistral AI raises $640M at $6B valuation by crazygringo on Tuesday June 11, 2024 @06:07PM (#40652171)
> Know that they are buying services below cost? I don't think so.

To the contrary, of course you should know.

If you know anything about tech businesses, you know that startups being funded by massive amounts of VC keep prices artificially low initially, and that they then rise.

This is tech startup 101 stuff.

It's fine if you're not aware of this as an engineer. But if you're building a business and therefore have a business model where you've identified known risks, there's no excuse for you not to be aware of it.

Re:Mistral AI raises $640M at $6B valuation by IanCal on Wednesday June 12, 2024 @02:08AM (#40655058)
I know that it happens, but that's different from knowing of it's happening here or not, and by how much.

Is openai selling below cost? Is the cost on azure lower than the true cost? ChatGPT free is obviously run subsidized but are the apis?

Can you tell me the actual cost to Microsoft of a gpt4 API call?

Re:Mistral AI raises $640M at $6B valuation by Eisenstein on Tuesday June 11, 2024 @05:30PM (#40651756)
Every business owner does not know what prices for other companies actually are -- they assume that they are being charged a price which has been determined by the market and is worthwhile to the seller to cover costs and profit.

By taking outside money and subsidizing certain market players so that they can sell their goods at below market value (because that is what is happening) you are undermining the market and punishing players who are acting rationally.

This would be absolutely no different than taking tax money and subsidizing certain companies over others to destroy competition and then claiming that it was a market decision.

'We are all adults here' is not a magical phrase that makes cheating OK.

Re:Mistral AI raises $640M at $6B valuation by simonw on Tuesday June 11, 2024 @05:18PM (#40651610)
Because they are platform companies.

If I launch a business that makes sense given the current pricing of LLMs, and the price then goes up by a factor of ten, my own unit economics may not make sense any more.

There's an Uber comparison here: back when Uber was cheap, some people made life decisions about where they lived and worked that were predicated on being able to afford a commute in an Uber (since public transport doesn't serve a lot of places, at least in the US).

This hurt a lot when the prices went up a few years later!

Re:Mistral AI raises $640M at $6B valuation by 0xcde4c3db on Tuesday June 11, 2024 @05:39PM (#40651879)
> If the service you use goes out of business, it's pretty easy to switch to another.

I'm not sure how well that works if vendors differentiate themselves through proprietary models and backend integrations. As far as I know, Mistral only releases weights for some of its models, with others being proprietary.

Re:Mistral AI raises $640M at $6B valuation by observationist on Tuesday June 11, 2024 @05:14PM (#40651570)
Investors don't know what they're doing. This is laughably apparent. They're monkeys throwing bananas at dart boards. There's that whole negative externality thing. There aren't alternatives to FAANG because they've viciously used their VC money to drive out competition, used their lawyers to litigate competition into extinction, and every other tool at their disposal to press an advantage in consumer exposure. This isn't a fair market where the best product wins; it's a game of "who's got the most money."

https://en.wikipedia.org/wiki/Enshittification

This is how the world gets shittier. It's a whole thing. It's a large part of why things are as bad as they are.

But hey, yeah, don't worry about it, they obviously know what they're doing.

Re:Mistral AI raises $640M at $6B valuation by __loam on Tuesday June 11, 2024 @05:23PM (#40651676)
Using capital to subsidize costs and sell your product for less than it's worth actually seems like a huge market failure to me, and is the consequence of huge concentrations of capital in the hands of a small and shrinking group of people. The only people who can afford to compete are the already wealthy. It stifles competition.

As a consumer it should definitely concern you if you're using poorly financed or unsustainable products. Maybe it's great now but I would rather have a slightly more expensive product that is more stable than one that suddenly had to make the experience a lot worse because they burned a lot of cash and had a down round.

Re:Mistral AI raises $640M at $6B valuation by LordDragonfang on Tuesday June 11, 2024 @05:25PM (#40651696)
>It's a well-established, economically rational business model to subsidize prices in order to gain market share.

Economically rational for the business owner, maybe, but it inevitably results in eventual enshittification for the consumer. (And Uber isn't a great example of this; its share price has only recently surpassed what it was at IPO 4 years ago, and many companies don't even do that well)

Re:Mistral AI raises $640M at $6B valuation by 6bb32646d83d on Wednesday June 12, 2024 @02:12PM (#40661131)
GPU companies providing inference on open source models (like deepinfra or togetherAI) are doing so at an extremely competitive cost, making me think that the API pricing of the big players right now is profitable.

(for example, deepinfra has wizardLM-2-8x22B at $0.65/1M output tokens, compared to $6/1M output tokens for 8x22B by Mistral - and of course Mistral has some more expensive, closed source models that perform better)