How Is LLaMa.cpp Possible?

Re:How Is LLaMa.cpp Possible? by arijun on Wednesday August 16, 2023 @04:09AM (#37144085)

Interesting, how would that work? Are there any well-known examples?

Is it: the weights all happen to be where float is sparse, so quantization ends up increasing fidelity? Or is it more of a “worse is better” dropout-type situation?

[ Parent | on HN ]

Re:How Is LLaMa.cpp Possible? by matsemann on Wednesday August 16, 2023 @04:40AM (#37144311)

For image recognition it can sometimes be like that. My gut feeling is that lowering from fp32 to fp16 can get rid of some kind of overfitting or so.

[ Parent | on HN ]

Re:How Is LLaMa.cpp Possible? by WithinReason on Wednesday August 16, 2023 @04:44AM (#37144344)

I suspect it works as regularisation of the network. It usually happens when you train with quantisation instead of post-training quantisation, an I haven't seen that done with LLMs yet.

[ Parent | on HN ]