News for hackers. Stuff that matters.
Re:Ollama releases OpenAI API compatibility by swyx on Thursday February 08, 2024 @04:14PM (#39307809)
I know a few people privately unhappy that openai api compatibility is becoming a community standard. Apart from some awkwardness around data.choices.text.response and such unnecessary defensive nesting in the schema, I don't really have complaints.

wonder what pain points people have around the API becoming a standard, and if anyone has taken a crack at any alternative standards that people should consider.

Re:Ollama releases OpenAI API compatibility by Patrick_Devine on Thursday February 08, 2024 @04:49PM (#39308202)
TBH, we debated about this a lot before adding it. It's weird being beholden to someone else's API which can dictate what features we should (or shouldn't) be adding to our own project. If we add something cool/new/different to Ollama will people even be able to use it since there isn't an equivalent thing in the OpenAI API?
Re:Ollama releases OpenAI API compatibility by minimaxir on Thursday February 08, 2024 @04:59PM (#39308350)
That's more of a marketing problem than a technical problem. If there is indeed a novel use case with a good demo example that's not present in OpenAI's API, then people will use it. And if it's really novel, OpenAI will copy it into their API and thus the problem is no longer an issue.

The power of open source!

Re:Ollama releases OpenAI API compatibility by Patrick_Devine on Thursday February 08, 2024 @05:16PM (#39308554)
You're right that it's a marketing problem, but it's also a technical problem. If tooling/projects are built around the compat layer it makes it really difficult to consume those features without having to rewrite a lot of stuff. It also places a cognitive burden on developers to know which API to use. That might not sound like a lot, but one of the guiding principles around the project (and a big part of its success) is to keep the user experience as simple as possible.
Re:Ollama releases OpenAI API compatibility by satellite2 on Thursday February 08, 2024 @05:22PM (#39308603)
At some point, (probably in a relatively close future), there will be the AI Consortium (AIC) to decide what enters the common API?
Re:Ollama releases OpenAI API compatibility by minimaxir on Thursday February 08, 2024 @04:17PM (#39307842)
That's why it's good as an option to minimize friction and reduce lock-in to OpenAI's moat.
Re:Ollama releases OpenAI API compatibility by simonw on Thursday February 08, 2024 @05:47PM (#39308864)
I want it to be documented.

I'm fine with it emerging as a community standard if there's a REALLY robust specification for what the community considers to be "OpenAI API compatible".

Crucially, that standard needs to stay stable even if OpenAI have released a brand new feature this morning.

So I want the following:

- A very solid API specification, including error conditions

- A test suite that can be used to check that new implementations conform to that specification

- A name. I want to know what it means when software claims to be "compatible with OpenAI-API-Spec v3" (for example)

Right now telling me something is "OpenAI API compatible" really isn't enough information. Which bits of that API? Which particular date-in-time was it created to match?

Re:Ollama releases OpenAI API compatibility by londons_explore on Thursday February 08, 2024 @06:17PM (#39309147)
It's a JSON API... JSON API's tend to be more... 'flexible'.

To consume them, just assume that every field is optional and extra fields might appear at any time.

Re:Ollama releases OpenAI API compatibility by swyx on Thursday February 08, 2024 @06:34PM (#39309300)
and disappear at any time... was a leetle bit unsettled by the sudden deprecation of "functions" for "tools" with only minor apparante benefit
Re:Ollama releases OpenAI API compatibility by nl on Thursday February 08, 2024 @06:51PM (#39309444)
and what does `auto` even mean?
Re:Ollama releases OpenAI API compatibility by athyuttamre on Friday February 09, 2024 @05:56AM (#39313572)
Hey nl — I work at OpenAI on our API. Do you mean `tool_choice="auto"`? If so, it means the model gets to pick which tool to call. The other options are:

- `tool_choice={type: "function", function: {name: "getWeather"}}`, where the developer can force a specific tool to be called. - `tool_choice="none"`, where the developer can force the model to address the user, rather than call a tool.

If you have any other feedback, please feel free to email me at [email protected]. Thanks!

Re:Ollama releases OpenAI API compatibility by athyuttamre on Friday February 09, 2024 @05:55AM (#39313561)
Hey swyx — I work at OpenAI on our API. Sorry the change was surprising, we definitely didn't do a great job communicating it.

To confirm, the `functions` parameter will continue to be supported.

We renamed `functions` to `tools` to better align with the naming across our products (Assistants, ChatGPT), where we support other tools like `code_interpreter` and `retrieval` in addition to `function`s.

If you have any other feedback for us, please feel free to email me at [email protected]. Thanks!

Re:Ollama releases OpenAI API compatibility by londons_explore on Friday February 09, 2024 @06:03AM (#39313613)
Might be a good idea to have API versions for this... Then when someone builds a product against "version 1", they can be sure that new features might be added to version 1, but no fields will be removed/renamed without openai releasing version 2.
Re:Ollama releases OpenAI API compatibility by te_chris on Friday February 09, 2024 @04:43AM (#39313173)
Amen! The lack of decent errors from OpenAI is the most annoying. They'll silently return 400 with no explanation. Let's hope that doesn't catch on.

OpenAI compatible just seems to mean 'you can format your prompt like the `messages` array'.

Re:Ollama releases OpenAI API compatibility by athyuttamre on Friday February 09, 2024 @05:52AM (#39313540)
Hi te_chris — I work at OpenAI and am currently working to improve our error messages. Would you be willing to share more about what errors you find annoying? My email is [email protected] (or feel free to reply here). Thanks!
Re:Ollama releases OpenAI API compatibility by tracerbulletx on Thursday February 08, 2024 @06:22PM (#39309193)
It's so trivially easy to create your own web server in your language of choice that calls directly into llama.cpp functions with the bindings for your language of choice it doesn't really matter all that much. If you want more control you can get with just a little more work. You don't really need these plug and play things.
Re:Ollama releases OpenAI API compatibility by sheepscreek on Thursday February 08, 2024 @06:24PM (#39309220)
I would take an imperfect standard over no standard any day!
Re:Ollama releases OpenAI API compatibility by dimask on Thursday February 08, 2024 @08:53PM (#39310474)
There is a difference between a standard and a monopoly, though.