Re:ChatGPT vs. open source on harder tasks
by andy_xor_andrew on Thursday May 18, 2023 @04:17PM
(#35993505)
I think there will always be some element of non-determinism with model output, regardless of how explicitly you define your rules.
I think regarding: "Make sure to use real numbers, not fractions."
It would be simple using Guidance to create some kind of validation function, which guidance would invoke after it gets the response from the model. In this case you would check if the number is a real number or not. And if not, the model would invoke again with a different seed.
Assuming the model gives a real number, as expected, ~90% of the time, you would only see retries very infrequently, and you could be pretty confident that your output will match what you expect.
(again, I'm not associated with this project, I've only scratched the surface playing with it locally, so this is just speculation)
Re:ChatGPT vs. open source on harder tasks
by thomasahle on Thursday May 18, 2023 @04:30PM
(#35993655)
You don't need non-determinism. If you know you are going to generate a number, just mask the output of the softmax layer to only decode number tokens. That's exactly what stuff like Guidance and Outlines (https://twitter.com/remilouf/status/1653765371273330688) do.
Re:ChatGPT vs. open source on harder tasks
by boywitharupee on Friday May 19, 2023 @08:21AM
(#36000941)
How do you mask the output of the softmax layer? The link you referenced doesn't provide specific details.
I thought the point of Microsoft Guidance was that we could specify a grammar or regular expression that the output must match?