I think the biggest thing pushing me away from OpenAI was they were subsidizing the chat experience much more than the API and this seems to reconcile that quite a bit. Quite simply OpenAI is sweetening the pot here too much for me to really ignore, this is a massively subsdizised service. I honestly don't feel the switching costs in the future will outweigh the benefits I'm getting now.
This is very early in the maturity cycle for this tech. The options that will be available for private inference and fine tuning, for cloud-gpu/timeshare inference and fine tuning, and for competing hosted solutions are going to vastly different as months go by. What looks like squeezing value out of OpenAI today might look a lot like technical debt and frustrating lock-in a year from now.
That's what they're hoping you chase after, and if your product is defined by this technology, maybe that's what you have to do. But if you're just thinking about feature opportunities for a more robust product, judiciousness could pay off better than rushing. For now.
Just wanted to highlight this as such a great, concise way to look at the Buy vs Build with pretty much any cloud service, thanks!
Once marketing gets in charge of product, it's doomed. And I can't think of a product startup that it hasn't happened to. Particularly with this type of growth, at some point, the suits start to out number the techies 10:1.
This is why openeness and healthy competition is primordial.
If you set money on fire -- eventually there's a time when you need to stop doing that.
Yes, OpenAI might be (we don't know how much) burning through their $5B capital/Azure credits now, but I think the `turbo` models are starting to addressing this as well. And $20/month from a large user base can also add up pretty quick.
> This is very early in the maturity cycle for this tech.
Think about what value you get out of the services and what migration might look like. If you are making simple completion or chat calls with a clever prompt, then migration will probably be trivial when the time comes. Those features are the commodity that everyone will be offering and you'll be able to shop around for the ideal solution as alternatives become competitive.
Alternately, if you're handing OpenAI a ton of data for them to opaquely digest for fine tuning with no egress tools, or having them accumulate lots of other critical data with no egress, you're obviously getting yourself locked in.
The more features you use, the more idiosyncratic those features are, the more non-transferable those features are, and the more deeply you integrate those features, the more risk you're taking. So you want to consider whether the reward is worth that risk.
Different projects will legitimately have different answers for that.
Time to market is more important, you build users you get an edge, you can swap models later on (as long as you own the data).
There's very little there there in most of these folks.
(For the few where there is, though, I agree with you.)
I use it instead of Bing Chat now for cases where I really need a search engine and Google is useless. Mainly because it's faster, but I also like not having to open another browser.
Getting access to this type of interaction data with (mostly) humans must be quite valuable asset.
OpenAI doesn't have some sort of egress feed for your database.
This type of post is actually more disruptive than the post you're replying to.
As is yours. The author of that comment has contact information in their information, why not unicast them?
> The paradox of tolerance states that if a society's practice of tolerance is inclusive of the intolerant, intolerance will ultimately dominate, eliminating the tolerant and the practice of tolerance with them. Karl Popper described it as the seemingly self-contradictory idea that, in order to maintain a tolerant society, the society must retain the right to be intolerant of intolerance.
To answer your question more succinctly, because the poster isn't the only person who will read these comments.
I would bet the poster you replied to had the same intent.
To prove it, I'll post Dr. Emily Bender's fantastic podcast about the problematic issues behind the current LLM wave. https://peertube.dair-institute.org/w/qpKuiNLTuHHMnvWGjnA2D8
(I think they are behind the most cogent critiques and worth knowing.)
That's what they're trying to incentivize, especically with being able to upload files for their own implementation of RAG. You're not getting the vector representation of those files back, and switching to another provider will require rebuilding and testing that infrastructure.
The developer experience is lacking vs. other vector database providers and the performance doesn't match those that prioritize performance rather than devex. You're also spending time writing plumbing around postgres that isn't really transferrable work.
For some people already in the ecosystem it will make sense.
You're thinking of traditional apps and APIs.
In an AI application, most of the work is in prompt engineering, not wiring up the API to your app. Prompts that work well for one model will fail horribly for another. People spend months refining their prompts before they're safe to share with users, and switching platforms will require doing most of that refinement over again.
Remember: a good model with a good prompt will generate bad outputs sometimes.
A bad model with a bad prompt will generate a good output sometimes.
That is simply a fact with these non deterministic models.
You have to do many iterations for each prompt to verify they are working correctly.
> I’ve not had much problems moving between LLMs…
If you want to move your prompts to a different model, you’re effectively replacing one:
f(prompt + seed) => output
With different black box implementation.
Unless you’re measuring the output over multiple iterations of (seed) and verifying your prompt still does the right thing, it’s actually very likely that what you’ve done if take an application with a known output space and converted it to an application with an unknown output space…
…that partially overlaps the original output space!
So it looks like it’s the same.
…but it isn’t, and the “isn’t” is in weird edge cases.
Unless you’re measuring that, you simply now have an app that does “eh, who knows?”
So yes. Porting is trivial if you don’t care if you have the same functionality.
…but reliably porting is much harder (or longer).
I think we're exiting the phase where people can launch an AI app and have people use it just because of the initial "wow factor" and moving into the phase where users will start churning and businesses will need to make sure that their AI agent is performing and they they understand how well it's performing.
BTW its much faster and cheaper to artive at a good prompt if you sample the model in deterministic mode (ie temperature=0)
By default you have to guess if the difference is due to the prompt change or due to the dice roll, as you’ve noticed, but you don’t need to!
This is degenerate (greedy) behaviour, and not representative of the what the prompt will behave like at a higher temperature.
(At least, that’s my understanding; it’s a complex topic but broadly speaking there no specific reason, as far as I’m aware, to expect that a particular combination of params/prompt is representative of any other combination of params/prompt for the same model; it may be, but it may not. Certainly on models like GPT4 it is not, for reasons that are not clear to anyone. So… take care with your prompt testing. setting temperature to 0 is basically meaningless unless you expect to use a temperature of 0 in production. The results you get from your prompts at temp 0 are not generally reflective of the results you will get at temp > 0).
I tried some Mistral variants with larger context windows, and had very poor results… the model would often offer either an empty completion or a nonsensical completion, even though the content fit comfortably within the context window, and I was placing a direct question either at the beginning or end, and either with or without an explanation of the task and the content. Large contexts just felt broken. There are so many ways that we are more than “two weeks” from the open source solutions matching what OpenAI offers.
And that’s to say nothing of how far behind these smaller models are in terms of accuracy or instruction following.
For now, 6-12 months behind also isn’t good enough. In the uncertain case that this stays true, then a year from now the open models could be perfectly adequate for many use cases… but it’s very hard to predict the progression of these technologies.
Regardless, where did you find 1.8T for GPT-4 Turbo? The Turbo model is the one with the 128K context size, and the Turbo models tend to have a much lower parameter count from what people can tell. Nobody outside of OpenAI even knows how many parameters regular GPT-4 has. 1.8T is one of several guesses I have seen people make, but the guesses vary significantly.
I’m also not convinced that parameter counts are everything, as your comment clearly implies, or that chinchilla scaling is fully understood. More research seems required to find the right balance: https://espadrine.github.io/blog/posts/chinchilla-s-death.ht...
Let's just agree it's 100x-300x more parameters, and let's assume the open ai folks are pretty smart and have a sense for the optimal number of tokens to train on.
Additionally explanations for the raw mathematics of log likelihoods and their loss ballparks.
Interesting low-level stuff. These researchers are the best of the best working for the company that can afford them working on the best models available.
Grab an 8K context model, tweak some internals and try to pass 32K context into it - it's still an 8K model and will go glitchy beyond 8K unless it's trained at higher context lengths.
Anthropic for example talk about the model's ability to spot words in the entire Great Gatsby novel loaded into context. It's a hint to how the model is trained.
Parameter counts are a unified metric, what seems to be important is embedding dimensionality to transfer information through the layers - and the layers themselves to both store and process the nuance of information.
It's very compelling and opens up a lot of use cases, so I've been keeping an eye out for advancements. However, inferencing on 4xA100s would be the target today for YaRN and 128K to get a reasonable token rate on their version of Mistral.
The docs say:
> By default, images are generated at standard quality, but when using DALL·E 3 you can set quality: "hd" for enhanced detail. Square, standard quality images are the fastest to generate.
This just rings hollow to me. We lost the fights for database portability, cloud portability, payments/billing portability, and other individual SaaS lock-in. I don't see why it'll be different this time around.
No we didn’t. There are viable on-prem alternatives or cross cloud alternatives for everything popular on the cloud.
Many companies did choose to hand their destiny over to cloud providers but lots didn’t.
OpenAI is currently refusing far more enterprises than these products could "lock-in" even with 100% stickiness.
Makes it unlikely this is about lock-in or fighting churn when arguably, the best advertisement for GPT-4 is comparing its raw results to any other LLM.
If you said their goal was fomenting FOMO, I'd buy it. Curious, though, when they'll let the FOMO fulfillment rate go up by accepting revenue for servicing that demand.
I'd argue the opposite. The new "Threads" interface in the OpenAI admin section lets you see exactly how it's interpreting input/output specifically to address the black box effect.
Lastly, you don’t even need any sort of database to keep track of threads and messages. The API is now stateful!
I think that most of these changes are exciting and make it a lot easier for people to get started. There is no doubt in my mind though that the API is now an even bigger blackbox, and lock-in is slightly increased depending on how you integrate with it.
As for lock in, agreed completely.
Apart from that, it's pretty much replaced 80% of my search engine usage, I can ask it to collate reviews for a product from reddit and other sites, get the critical reception of a book, etc. You don't have to go and read long posts and articles, have GPT do it for you. There's many other use cases like this. For the second part, I'm using a UI called Typing Mind (which also works with the API).
That's cool!
> it's pretty much replaced 80% of my search engine usage
That's not cool. That's how you end up relying on nonexisting sources or other hallucinations.
That aside, this particular admonishment was worn out a couple of months after ChatGPT was released. It does not need to be repeated every time someone mentions doing something interesting with an LLM.
I have integrated a search engine plugin and a web browsing plugin, which means I don't have to do the search, for example I can ask it to compare the battery life of 3 phones, it'll do 3 searches, might open couple of reddit threads too, then give me the info that I need. It's miles ahead of the current experience with search engines.
“Derp derp, hallucinations”.
Eh, no, not in practice, not when the entire context and document is provided and the tools are used correctly.
Not everything is just data in a database or some structured format. Sometimes you have blobs of text from a user, or maybe you ran whisper on an audio/video file and now you just have a transcript blob… it’s never been easier to automate all of this stuff and get accurate results.
You can even have humans in the loop still to protect against hallucinations, or use one model to validate another (ask GPT to correct or flag issues with a whisper transcript)
Getting a chiding lecture every time you ask an AI to do something does absolutely nothing for the end user other than waste their time. "AI Safety" academics are memeing themselves out of the future of this tech and leaving the gate wide open for "unsafe" AI to flourish with this farcical behavior.
I'm mixed on the presentation and will need to read the fine print on the API docs on all of these things, which have been updated just now: https://platform.openai.com/docs/api-reference
The pricing page has now updated as well: https://openai.com/pricing
Notably, the DALL-E 3 API is $0.04 per image which is an order of magnitude above everyone else in the space.
EDIT: One interesting observation with the new OpenAI pricing structure not mentioned during the keynote: finetuned ChatGPT 3.5 is now 3x of the cost of the base ChatGPT 3.5, down from 8x the cost. That makes finetuning a more compelling option.