News for hackers. Stuff that matters.
Re:GPT-3.5 Turbo fine-tuning and API updates by steno132 on Tuesday August 22, 2023 @08:46PM (#37230522)
And OpenAI continues to ship.

As promised, they released GPT3.5 fine-tuning today. They opened GPT4 API access a few months ago. In a few months, they'll release GPT4 fine-tuning.

Many favor open source AI, and criticize OpenAI for not being open enough. But the most important thing is, OpenAI innovates. Fast.

Llama, Bard, FB's open source stuff is good but it's lightyears behind OpenAI. You have to credit them for that.

Re:GPT-3.5 Turbo fine-tuning and API updates by syntaxing on Tuesday August 22, 2023 @08:54PM (#37230569)
Prior to LLaMA 2, I would have agreed with you but LLaMA 2 is a game changer. The 70B performance is probably between 3.5 and 4. But running it personally isn't cheap. The cheapest I found is about $4/hr to run the whole thing. I only spend around $3 on average a month on GPT-3.5 API for my personal stuff.
Re:GPT-3.5 Turbo fine-tuning and API updates by jimmcslim on Tuesday August 22, 2023 @09:35PM (#37230836)
Out of curiosity and if you are happy to share, what is your 'personal stuff'?
Re:GPT-3.5 Turbo fine-tuning and API updates by SOLAR_FIELDS on Tuesday August 22, 2023 @09:37PM (#37230845)
As a counter reference, for my work I use it to code (for-4) and it has been between $70 and $200 per month depending on how heavily I use it
Re:GPT-3.5 Turbo fine-tuning and API updates by syntaxing on Tuesday August 22, 2023 @10:25PM (#37231190)
GPT-4 is significantly more expensive so I can definitely see you spending that amount. For really complex stuff, I switch over the GPT-4 and it will cost me almost $3 a "question" (as in going from the beginning to solving it). Honestly worth it since it solves my problem but it adds up quick so I try to stick with 3.5 when I can.
Re:GPT-3.5 Turbo fine-tuning and API updates by blorenz on Tuesday August 22, 2023 @10:57PM (#37231362)
Can’t you get by with ChatGPT-4 for these personal assistant type questions? That’s what I do and my 20 a month goes a long way. I’d be interested to see if I am missing out on anything using GPT to is way in contrast to the API.
Re:GPT-3.5 Turbo fine-tuning and API updates by SOLAR_FIELDS on Wednesday August 23, 2023 @12:52AM (#37232006)
I use it with a tool that is wired into my terminal that changes my files for me [1]. That alone makes me several times more productive compared to copy pasting back and forth between the chat window. If the chat window makes me twice as productive the command line tool probably makes me 5x as productive. At that kind of output on a developer salary the $70-200 a month is absolute peanuts compared to what you get in return

1: https://github.com/paul-gauthier/aider

Re:GPT-3.5 Turbo fine-tuning and API updates by Jach on Wednesday August 23, 2023 @01:06AM (#37232084)
Is your code subject to code review? If so have you done anything to improve that bottleneck, or was it never an issue at previous productivity?
Re:GPT-3.5 Turbo fine-tuning and API updates by SOLAR_FIELDS on Wednesday August 23, 2023 @12:06PM (#37238164)
It is subject to code review, but I typically spent much more time writing code than having it reviewed (I am very methodical and slow writing code)
Re:GPT-3.5 Turbo fine-tuning and API updates by blorenz on Thursday August 24, 2023 @08:23AM (#37247963)
This tool looks splendid. Personally, it evokes in me the memories of MUDding back in the early 90s. What a concept that would be to MUD to build apps via LLM -- or even MUD to build the MUD in real-time outside of the OLC and scripting. That sounds like a passion project for me when I can find the time.
Re:GPT-3.5 Turbo fine-tuning and API updates by syntaxing on Wednesday August 23, 2023 @07:14AM (#37234178)
I actually used to use ChatGPT but switched to the API once I had GPT-4 access. Mainly it’s because I simply didn’t use the $20 worth of the GPT-4 at the time. It was extremely slow and the question per hour limitation was annoying and stressful. I would always worry I would need it for something unexpected so I never used more than 15 questions at a time (but this has probably changed these couple months). In addition, the privacy implications are better for the API since the terms are better for how they handle your data. I also like how I can tie in GPT anywhere. I use the matrix bridge so you can give access to people like my parents who are not as tech literate to sign up and get used to chatgpt interface. I allow them to talk to it as a bot through WhatsApp bridge.
Re:GPT-3.5 Turbo fine-tuning and API updates by syntaxing on Tuesday August 22, 2023 @10:20PM (#37231155)
I use it a lot for personal coding projects, grammar correction/sentence rewording, and translation (it works better than google translate for longer text). I explicitly call out personal stuff since my job provides an in-house front end that uses the GPT API (I'm actually not sure which version it is, but guessing from the response quality, its probably GPT-4). My work one has made me noticeably more productive. It helps me with a lot of the "boring" work that I procrastinate a lot on. It starts my momentum and allows me focus a lot on the "complex stuff". I'm not sure how much money I use since there is no limit at work but if I had to guess, its probably north of $100 a month on credit.
Re:GPT-3.5 Turbo fine-tuning and API updates by manjoh on Wednesday August 23, 2023 @12:56AM (#37232030)
Can you talk about how you integrate gpt API at work and why not just use chatgpt 4?
Re:GPT-3.5 Turbo fine-tuning and API updates by syntaxing on Wednesday August 23, 2023 @07:07AM (#37234135)
The server is provided by my employer so I can’t go into the implementation detail. But overall, most companies provide access to the API endpoint instead of using chatGPT itself since OpenAI uses your results to train (hence why it’s free for 3.5). The API endpoint supposedly doesn’t use your data for training which is why I use the API endpoint for personal stuff as well.
Re:GPT-3.5 Turbo fine-tuning and API updates by armini on Tuesday August 22, 2023 @09:35PM (#37230835)
How are you currently hosting your LLaMA 2? Any tips, tricks or advice?
Re:GPT-3.5 Turbo fine-tuning and API updates by syntaxing on Tuesday August 22, 2023 @10:35PM (#37231249)
It depends on your needs. For instance, do you want to host an API or do you want to have a front end like chatGPT? Chances are, text-generation-webui [1] should get you pretty close to hosting it yourself. You simply clone the repo, download the model from huggingface using the included helper (download-model.py) and fire up the server with server.py. You can connect to it by SSH port tunneling on port 7860 (there's other way like Ngrok but SSH tunneling is the easiest and secure).

As for hosting, I found that runpod [2] has been the cheapest (not affiliated, just a user). All the other services tend to add up more than them when you include bandwidth and storage. There's some tutorials online [3] but a lot of them use the quantized version. You should be able to fit the original 70B with "load_in_8bit" on one A100 80GB.

[1] https://github.com/oobabooga/text-generation-webui [2] https://www.runpod.io/ [3] https://gpus.llm-utils.org/running-llama-2-on-runpod-with-oo...

Re:GPT-3.5 Turbo fine-tuning and API updates by robertnishihara on Wednesday August 23, 2023 @12:46AM (#37231980)
If you want to query the Llama-2 models, you can use Anyscale Endpoints [1]. Note: I work on this :)

Llama-2-70B is $1 / million tokens, which is the most cost-efficient on the market that I'm aware of.

[1] https://app.endpoints.anyscale.com/

Re:GPT-3.5 Turbo fine-tuning and API updates by zo1 on Wednesday August 23, 2023 @01:41AM (#37232249)
Can we supply our own fine-tuned models?
Re:GPT-3.5 Turbo fine-tuning and API updates by andrewmunn on Wednesday August 23, 2023 @01:03AM (#37232069)
How do you keep the cost down?
Re:GPT-3.5 Turbo fine-tuning and API updates by TeMPOraL on Wednesday August 23, 2023 @02:20PM (#37240096)
Tried to plug it in to my favorite chat frontend (TypingMind), bounced off CORS. Is this something you can do something about?
Re:GPT-3.5 Turbo fine-tuning and API updates by easygenes on Tuesday August 22, 2023 @11:58PM (#37231689)
For what tasks do you consider 70B beyond GPT-3.5 performance? There are some I’m aware of, but they are very much the exception and not the rule, even with the best 70B fine-tunes currently available.
Re:GPT-3.5 Turbo fine-tuning and API updates by syntaxing on Wednesday August 23, 2023 @07:04AM (#37234117)
I mainly use 70B for “text QA” on files I find sensitive like personal documents. The answers have been very close to what I get if I use GPT-3 (langchain makes it easy to switch). Do you use the quantized version? If so, try running the full one on a A100.
Re:GPT-3.5 Turbo fine-tuning and API updates by ozr on Wednesday August 23, 2023 @12:56AM (#37232031)
I run 70B very cheaply using serverless GPUs. I've had the best experience with Runpod, but there are a few other options out there for it as well.
Re:GPT-3.5 Turbo fine-tuning and API updates by victor106 on Tuesday August 22, 2023 @11:57PM (#37231676)
>And OpenAI continues to ship.

This.

Google (specifically their CEO) was saying since at least 2016 that "Google is an AI first company". (Whatever the hell that means). But they had no product to show for and they are on the verge of being the next IBM.

Re:GPT-3.5 Turbo fine-tuning and API updates by pocket_cheese on Wednesday August 23, 2023 @01:33AM (#37232216)
I have lots of criticisms of Google, but let's not forget they have Gmail + workspaces,Google Domains, Android, Google Cloud, Google Fiber, Google Fi, The pixel, Nest, Maps, Youtube, Fitbit, and probably the browser 90% of people on here are using. Would losing search devastate them? Absolutely, but they're not exactly IBM levels of irrelevance without it.
Re:GPT-3.5 Turbo fine-tuning and API updates by abatilo on Wednesday August 23, 2023 @01:49AM (#37232284)
Search is the majority of their revenue so that they can subsidize their other products. Devastation somehow doesn't feel like it properly captures how much it would hurt them to lose it.
Re:GPT-3.5 Turbo fine-tuning and API updates by predmijat on Wednesday August 23, 2023 @02:08AM (#37232369)
Google Domains is sold to Sqaurespace and at least 3 of the things you mentioned are a drop in the ocean.
Re:GPT-3.5 Turbo fine-tuning and API updates by Manheim on Wednesday August 23, 2023 @01:46AM (#37232274)
Well, it could mean that they are focused on AI development and utilizing AI in all of their products and services. Being an AI-first company does not necessarily mean that you build AI tools for others to use.

Google is lagging behind in the market space for public AI tools, agree, but I am not convinced they are as far behind in AI development as you indicate.

Re:GPT-3.5 Turbo fine-tuning and API updates by TeMPOraL on Wednesday August 23, 2023 @02:00PM (#37239812)
> Google (specifically their CEO) was saying since at least 2016 that "Google is an AI first company". (Whatever the hell that means).

What it means is it's why so many things about Google experience suck so badly. Whether it's what he meant or not, the practical flip side of Google being "AI first company" is that they're "humans last" company. Or, it's a different way of saying they only do things that scale. Telemetry and automated decision making scale. Human review and customer support do not.

Re:GPT-3.5 Turbo fine-tuning and API updates by ramraj07 on Wednesday August 23, 2023 @12:09AM (#37231770)
And they have 700 employees. Just that's it. Incredible.
Re:GPT-3.5 Turbo fine-tuning and API updates by andromeduck on Wednesday August 23, 2023 @12:37AM (#37231927)
The number of Google employees performing a similar function is probably not so different.
Re:GPT-3.5 Turbo fine-tuning and API updates by greatpostman on Wednesday August 23, 2023 @12:37AM (#37231928)
I love how far behind Facebook and google are. I’ve always said they weren’t hiring the best, but rather are monopolies sucking the air out of the room. This proves it.
Re:GPT-3.5 Turbo fine-tuning and API updates by nacs on Wednesday August 23, 2023 @12:45AM (#37231973)
How is Facebook behind?

They've released the most powerful open source LLM models so far (Llama 1 + 2) and are a serious threat to the Openai closed-source monopoly.

Re:GPT-3.5 Turbo fine-tuning and API updates by greatpostman on Wednesday August 23, 2023 @01:10AM (#37232096)
You can’t use llama2 like you can GPT4, it’s not even close.
Re:GPT-3.5 Turbo fine-tuning and API updates by justapassenger on Wednesday August 23, 2023 @01:25AM (#37232180)
You’re comparing apples to oranges.
Re:GPT-3.5 Turbo fine-tuning and API updates by huijzer on Wednesday August 23, 2023 @02:14AM (#37232405)
Still I was promised immense productivity by their tools, but I do keep seeing bugs in the ChatGPT interface. So they themselves seem not so productive? When seeing their tools as slightly productivity increasing, then they are good.