News for hackers. Stuff that matters.
Re:Comparing 60 LLMs with a set of 20 prompts by ossopite on Saturday September 09, 2023 @06:44PM (#37451054)
for performance reasons, yes, I believe it's because the accumulation is over parallel computations so the ordering is at the mercy of the scheduler. but I'm not familiar with the precise details

edit: at 13:42 in https://www.youtube.com/watch?v=TB07_mUMt0U&t=13m42s there is an explanation of the phenomenon in the context of training but I suspect the same kind of operation is happening during inference

There are no comments on this comment yet.