News for hackers. Stuff that matters.
Re:ChatGPT vs. open source on harder tasks by tikkun on Thursday May 18, 2023 @02:06PM (#35991823)
I wish the comparison was with GPT-4 instead of GPT-3.5.

When comparing model intelligence, I wish that only the most capable models were compared.

Re:ChatGPT vs. open source on harder tasks by furyofantares on Thursday May 18, 2023 @02:25PM (#35992071)
I agree, I only use 4 because by far the thing that matters most for my usage is the quality of the output at this stage in the game.

But 3.5 is generally available, fast, and costs a WHOLE LOT less for API users; those are real capabilities that matter though by "capable" you only meant the quality of the output. It's worth comparing to 3.5 also.

Re:ChatGPT vs. open source on harder tasks by doctoboggan on Thursday May 18, 2023 @02:59PM (#35992521)
At first I was skeptical of the difference between 3.5 and 4 but there really is a large gap in the models understanding of complex issues between them. I make a habit of always selecting gpt4 but forgot to yesterday. Within 3 responses I could immediately tell I was interacting with a less capable model.
Re:ChatGPT vs. open source on harder tasks by icelancer on Thursday May 18, 2023 @04:52PM (#35993886)
It really is tremendous for code fuzzing/optimizing. When I hit the rate limit on ChatGPT and switch to 3.5-turbo, the quality difference is immediately noticeable, so much so that I finally set up my own private ChatGPT instance with an API key for GPT-4 so I stopped running into these issues while coding.
Re:ChatGPT vs. open source on harder tasks by TeMPOraL on Thursday May 18, 2023 @07:23PM (#35995707)
Ever since I got API access to GPT-4, I don't even bother with using ChatGPT proper (I need to cancel the subscription one of these days). I tend to use it in bursts, so smoothing usage by rate limits doesn't work for me.

Is it more expensive this way? At the rate I'm using it, probably 2-3x. Is it worth it? Absolutely. I've never in my life felt so happy to pay for something per request.

Re:ChatGPT vs. open source on harder tasks by icelancer on Thursday May 18, 2023 @08:28PM (#35996285)
Yeah, I gotta just do that as well - convert to using GPT-4 entirely by rate and by API.

I use the plugins and new features on ChatGPT though, and it's nice to have a single interface to use 3.5-turbo for comparison, Code Interpreter once in awhile, etc. So I'll probably just keep it. It's $20/month, after all.

Re:ChatGPT vs. open source on harder tasks by cocoflunchy on Thursday May 18, 2023 @05:14PM (#35994139)
Small tip, you can bookmark https://chat.openai.com/?model=gpt-4 to avoid the click
Re:ChatGPT vs. open source on harder tasks by doctoboggan on Thursday May 18, 2023 @09:32PM (#35996702)
I actually use it through the API playground and not the chat interface. I like the explicit control it gives me over the context and system prompts.
Re:ChatGPT vs. open source on harder tasks by TeMPOraL on Thursday May 18, 2023 @07:12PM (#35995602)
Same here, though GPT-3.5 surprised me positively. I felt something was off, but didn't realize what, until I unexpectedly hit the context window limit - at which point it dawned on me that I forgot to switch to GPT-4... I didn't notice earlier, because GPT-3.5 managed to give solid answers to a relatively simple programming task I gave it (generating a single-page, dependency-free image gallery I could just drop on a server as HTML file and have it work).
Re:ChatGPT vs. open source on harder tasks by yokoprime on Thursday May 18, 2023 @02:58PM (#35992512)
GPT-3.5 is good enough for many cases. I know Microsoft for their Bing chat implementation is using a pre-prosessor to select the most appropriate model to use.
Re:ChatGPT vs. open source on harder tasks by micromacrofoot on Thursday May 18, 2023 @04:00PM (#35993310)
4 is much better than 3.5 at harder tasks though
Re:ChatGPT vs. open source on harder tasks by M4v3R on Thursday May 18, 2023 @03:20PM (#35992771)
Me too. GPT-4 is much much better at following the given instructions to the letter. If you specify a format you want to receive your response in it will almost always follow that format, while 3.5 will very often fail to do so.

Edit: out of curiosity I posted the meeting summarization example to GPT-4 and indeed it produced both the relevant snippets and the correct answer to the stated question.

Re:ChatGPT vs. open source on harder tasks by zellyn on Thursday May 18, 2023 @03:45PM (#35993096)
The authors share your wish:

> While Vicuna is somewhat comparable to ChatGPT (3.5), we believe GPT-4 is a _much_ stronger model, and are excited to see if open source models can approach _that_.