But 3.5 is generally available, fast, and costs a WHOLE LOT less for API users; those are real capabilities that matter though by "capable" you only meant the quality of the output. It's worth comparing to 3.5 also.
Is it more expensive this way? At the rate I'm using it, probably 2-3x. Is it worth it? Absolutely. I've never in my life felt so happy to pay for something per request.
I use the plugins and new features on ChatGPT though, and it's nice to have a single interface to use 3.5-turbo for comparison, Code Interpreter once in awhile, etc. So I'll probably just keep it. It's $20/month, after all.
Edit: out of curiosity I posted the meeting summarization example to GPT-4 and indeed it produced both the relevant snippets and the correct answer to the stated question.
> While Vicuna is somewhat comparable to ChatGPT (3.5), we believe GPT-4 is a _much_ stronger model, and are excited to see if open source models can approach _that_.
When comparing model intelligence, I wish that only the most capable models were compared.