News for hackers. Stuff that matters.
Re:New models and developer products announced at OpenAI DevDay by ebiester on Monday November 06, 2023 @02:08PM (#38167259)
I don't understand the lock-in argument here. Yes, if a competitor comes in there will be switching cost as everything is re-learned. However, from a code perspective, it is a function of the key and a relatively small API. New regulations outstanding, what is stoping someone from moving from OpenAI to Anthropic (for example) other than the cost of learning how to effectively utilize Anthropic for your use case?

OpenAI doesn't have some sort of egress feed for your database.

Re:New models and developer products announced at OpenAI DevDay by pclmulqdq on Monday November 06, 2023 @03:04PM (#38168132)
I sometimes wonder how much OpenAI pays for people to post arguments about how great they are on HN, because it looks like you are pretty much right. There isn't a ton about OpenAI that is actually sticky.
Re:New models and developer products announced at OpenAI DevDay by minimaxir on Monday November 06, 2023 @03:08PM (#38168201)
I most definitely am not paid by OpenAI and am very confused how my original (critical) comment could be seen as astroturfing.
Re:New models and developer products announced at OpenAI DevDay by airstrike on Monday November 06, 2023 @03:50PM (#38168808)
> Please don't post insinuations about astroturfing, shilling, brigading, foreign agents, and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email [email protected] and we'll look at the data.

https://news.ycombinator.com/newsguidelines.html

Re:New models and developer products announced at OpenAI DevDay by PH95VuimJjqBqy on Monday November 06, 2023 @04:17PM (#38169153)
likewise, stop posting these references to the guidelines, if you feel the guidelines are being broken report it to the moderators and move on.

This type of post is actually more disruptive than the post you're replying to.

Re:New models and developer products announced at OpenAI DevDay by 0x0000000 on Monday November 06, 2023 @04:37PM (#38169376)
> This type of post is actually more disruptive than the post you're replying to.

As is yours. The author of that comment has contact information in their information, why not unicast them?

Re:New models and developer products announced at OpenAI DevDay by PH95VuimJjqBqy on Monday November 06, 2023 @04:46PM (#38169468)
https://en.wikipedia.org/wiki/Paradox_of_tolerance

> The paradox of tolerance states that if a society's practice of tolerance is inclusive of the intolerant, intolerance will ultimately dominate, eliminating the tolerant and the practice of tolerance with them. Karl Popper described it as the seemingly self-contradictory idea that, in order to maintain a tolerant society, the society must retain the right to be intolerant of intolerance.

To answer your question more succinctly, because the poster isn't the only person who will read these comments.

Re:New models and developer products announced at OpenAI DevDay by 0x0000000 on Tuesday November 07, 2023 @08:08AM (#38176300)
> To answer your question more succinctly, because the poster isn't the only person who will read these comments.

I would bet the poster you replied to had the same intent.

Re:New models and developer products announced at OpenAI DevDay by PH95VuimJjqBqy on Tuesday November 07, 2023 @10:08AM (#38177616)
I'll always defend your right to say whatever you want but that never implies it's valid.
Re:New models and developer products announced at OpenAI DevDay by ebiester on Monday November 06, 2023 @04:17PM (#38169158)
I have no financial vested interest.

To prove it, I'll post Dr. Emily Bender's fantastic podcast about the problematic issues behind the current LLM wave. https://peertube.dair-institute.org/w/qpKuiNLTuHHMnvWGjnA2D8

(I think they are behind the most cogent critiques and worth knowing.)

Re:New models and developer products announced at OpenAI DevDay by minimaxir on Monday November 06, 2023 @03:11PM (#38168250)
> OpenAI doesn't have some sort of egress feed for your database.

That's what they're trying to incentivize, especically with being able to upload files for their own implementation of RAG. You're not getting the vector representation of those files back, and switching to another provider will require rebuilding and testing that infrastructure.

Re:New models and developer products announced at OpenAI DevDay by goosinmouse on Monday November 06, 2023 @05:36PM (#38170097)
Thats exactly what i thought. Smart strategy on OpenAI's part given that its extremely easy (and free) to do RAG with pgvector.
Re:New models and developer products announced at OpenAI DevDay by nostrebored on Tuesday November 07, 2023 @10:08AM (#38177611)
It's neither free nor performant.

The developer experience is lacking vs. other vector database providers and the performance doesn't match those that prioritize performance rather than devex. You're also spending time writing plumbing around postgres that isn't really transferrable work.

For some people already in the ecosystem it will make sense.

Re:New models and developer products announced at OpenAI DevDay by lolinder on Monday November 06, 2023 @06:22PM (#38170678)
> However, from a code perspective, it is a function of the key and a relatively small API.

You're thinking of traditional apps and APIs.

In an AI application, most of the work is in prompt engineering, not wiring up the API to your app. Prompts that work well for one model will fail horribly for another. People spend months refining their prompts before they're safe to share with users, and switching platforms will require doing most of that refinement over again.

Re:New models and developer products announced at OpenAI DevDay by Aeolun on Monday November 06, 2023 @06:45PM (#38170961)
I’d be more worried about this if OpenAI had a track record of increasing prices, but the opposite happens. I get more for the same price basically every 6 months.
Re:New models and developer products announced at OpenAI DevDay by jairuhme on Monday November 06, 2023 @07:28PM (#38171376)
I'd imagine that will start to switch back the other way at some point. Decrease prices to gain market share and get you locked in, then increase prices to earn more money to keep VC's happy
Re:New models and developer products announced at OpenAI DevDay by dom96 on Monday November 06, 2023 @06:51PM (#38171041)
Sure, they are decreasing the prices right now. But once it comes time for them to become profitable they can easily reverse course.
Re:New models and developer products announced at OpenAI DevDay by edgyquant on Monday November 06, 2023 @07:15PM (#38171244)
Switching from one API to another generally requires refactoring. I’ve not had much problems moving between LLMs (openai to Anthropic)
Re:New models and developer products announced at OpenAI DevDay by wokwokwok on Monday November 06, 2023 @07:31PM (#38171413)
Then you’re either not testing your prompts or doing something trivial.

Remember: a good model with a good prompt will generate bad outputs sometimes.

A bad model with a bad prompt will generate a good output sometimes.

That is simply a fact with these non deterministic models.

You have to do many iterations for each prompt to verify they are working correctly.

> I’ve not had much problems moving between LLMs…

If you want to move your prompts to a different model, you’re effectively replacing one:

f(prompt + seed) => output

With different black box implementation.

Unless you’re measuring the output over multiple iterations of (seed) and verifying your prompt still does the right thing, it’s actually very likely that what you’ve done if take an application with a known output space and converted it to an application with an unknown output space…

…that partially overlaps the original output space!

So it looks like it’s the same.

…but it isn’t, and the “isn’t” is in weird edge cases.

Unless you’re measuring that, you simply now have an app that does “eh, who knows?”

So yes. Porting is trivial if you don’t care if you have the same functionality.

…but reliably porting is much harder (or longer).

Re:New models and developer products announced at OpenAI DevDay by esafak on Monday November 06, 2023 @11:11PM (#38173160)
How many people are even writing tests for these things?
Re:New models and developer products announced at OpenAI DevDay by dbmikus on Monday November 06, 2023 @11:48PM (#38173365)
Very few. Many deployed apps don't have a good quantitative grasp of the quality of their LLMs. Some are doing testing or evaluation, through things like unit tests, A/B testing different prompts, collecting user feedback.

I think we're exiting the phase where people can launch an AI app and have people use it just because of the initial "wow factor" and moving into the phase where users will start churning and businesses will need to make sure that their AI agent is performing and they they understand how well it's performing.

Re:New models and developer products announced at OpenAI DevDay by wizzard0 on Tuesday November 07, 2023 @02:26AM (#38174137)
> many iterations of each prompt

BTW its much faster and cheaper to artive at a good prompt if you sample the model in deterministic mode (ie temperature=0)

By default you have to guess if the difference is due to the prompt change or due to the dice roll, as you’ve noticed, but you don’t need to!

Re:New models and developer products announced at OpenAI DevDay by wokwokwok on Tuesday November 07, 2023 @03:57AM (#38174680)
You should have a read of https://huggingface.co/blog/how-to-generate (the section on sampling, with regard to setting temperature to zero).

This is degenerate (greedy) behaviour, and not representative of the what the prompt will behave like at a higher temperature.

(At least, that’s my understanding; it’s a complex topic but broadly speaking there no specific reason, as far as I’m aware, to expect that a particular combination of params/prompt is representative of any other combination of params/prompt for the same model; it may be, but it may not. Certainly on models like GPT4 it is not, for reasons that are not clear to anyone. So… take care with your prompt testing. setting temperature to 0 is basically meaningless unless you expect to use a temperature of 0 in production. The results you get from your prompts at temp 0 are not generally reflective of the results you will get at temp > 0).

Re:New models and developer products announced at OpenAI DevDay by Rastonbury on Monday November 06, 2023 @09:57PM (#38172639)
Still moving between models is less arduous than switching cloud providers, depending on use case and price difference of course. Most models hold GPT4 as the benchmark they aspire to and should converge to its capabilities.
Re:New models and developer products announced at OpenAI DevDay by lolinder on Tuesday November 07, 2023 @09:44AM (#38177313)
It's not a question of converging on its capabilities, it's a question of responding equivalently to the nuances of the prompt you've crafted. Two models can be equally capable, but one might interpret a phrase in your prompt slightly differently.