News for hackers. Stuff that matters.
Re:ChatGPT vs. open source on harder tasks by thomasahle on Thursday May 18, 2023 @04:30PM (#35993655)
You don't need non-determinism. If you know you are going to generate a number, just mask the output of the softmax layer to only decode number tokens. That's exactly what stuff like Guidance and Outlines (https://twitter.com/remilouf/status/1653765371273330688) do.
Re:ChatGPT vs. open source on harder tasks by boywitharupee on Friday May 19, 2023 @08:21AM (#36000941)
How do you mask the output of the softmax layer? The link you referenced doesn't provide specific details.