Re:GPT-4 and professional benchmarks: the wrong answer to th...
by anon7725 on Tuesday March 21, 2023 @10:32AM
(#35246737)
The parent said:
> I'm at the point where I'm preparing to integrate models all over the place
Nobody understands these models right now. We don’t even have the weights.
You may draw some artificial distinction between literally checking in the source code of a model into your git repo and making a call to some black box API that hosts it. And you may claim that doing so is no different than making a call to Twilio or whatever, but I think there is a major difference: nobody can make a claim about what an LLM will return or how it will return it, cannot make guarantees about how it will fail, etc.
> I'm at the point where I'm preparing to integrate models all over the place
Nobody understands these models right now. We don’t even have the weights.
You may draw some artificial distinction between literally checking in the source code of a model into your git repo and making a call to some black box API that hosts it. And you may claim that doing so is no different than making a call to Twilio or whatever, but I think there is a major difference: nobody can make a claim about what an LLM will return or how it will return it, cannot make guarantees about how it will fail, etc.
I agree with zer00eyz.