Re:Learning to Reason with LLMs
by RobertDeNiro on Thursday September 12, 2024 @07:08PM
(#41526596)
Also seems very impractical to embed this into a deployed product. How can you possibly hope to control and estimate costs? I guess this is strictly meant for R&D purposes.
[ Parent
| on HN ]
Re:Learning to Reason with LLMs
by sebzim4500 on Thursday September 12, 2024 @07:18PM
(#41526659)
You can specify the max length of the response, which presumably includes the hidden tokens.
I don't see why this is qualitatively different from a cost perspective than using CoT prompting on existing models.
You can’t verify that you’re paying what you should be if you can’t see the hidden tokens.
Re:Learning to Reason with LLMs
by sebzim4500 on Friday September 13, 2024 @11:41AM
(#41532239)
With the conventional models you don't get the activations or the logits even though those would be useful.
Ultimately if the output of the model is not worth what you end up paying for it then great, I don't see why it really matters to you whether OpenAI is lying about token counts or not.
As a single user, it doesn’t really, but as a SaaS operator I want tractable, hopefully predictable pricing.
I wouldn’t just implicitly trust a vendor when they say “yeah we’re just going to charge you for what we feel like when we feel like. You can trust us.”
Re:Learning to Reason with LLMs
by BoorishBears on Thursday September 12, 2024 @08:27PM
(#41527036)
For one, you don't get to see any output at all if you run out of tokens during thinking.
If you set a limit, once it's hit you just get a failed request with no introspection on where and why CoT went off the rails
Why would I pay for zero output? That’s essentially throwing money down the drain.
Re:Learning to Reason with LLMs
by HarHarVeryFunny on Thursday September 12, 2024 @10:27PM
(#41527552)
They are currently trying to raise money (talk of new $150B valuation), so that may have something to do with it