News for hackers. Stuff that matters.
Re:Language Models Are Super Mario: Absorbing Abilities from... by visarga on Saturday April 06, 2024 @04:25PM (#39955194)
> For example instead of training an 8x7b mixture of experts then merging, just incorporate the sparsity constraint while pre-training a single 7b model (somehow).

I'm thinking it would help reduce the network demands for gradient updates if merge from time to time. That could unlock distributed training, like SETI@Home.

There are no comments on this comment yet.