Automatically Detecting Under-Trained Tokens in Large Language Models

Re:Automatically Detecting Under-Trained Tokens in Large Lan... by londons_explore on Sunday May 12, 2024 @09:34AM (#40334329)

We shouldn't just be looking for under trained tokens. Tokens are effectively the first layer of the network, but we should also be looking for training data imbalances at every weight at every other layer of the network.

When we find them, it might be best to delete weights with hardly any data flowing through them (which might make the model smaller or help generalisation).

[ Parent | on HN ]

Re:Automatically Detecting Under-Trained Tokens in Large Lan... by dissuade on Sunday May 12, 2024 @04:15PM (#40337259)

We can already compress and/or merge holomorphic models.

[ Parent | on HN ]

Re:Automatically Detecting Under-Trained Tokens in Large Lan... by mycall on Sunday May 12, 2024 @11:28PM (#40339541)

> delete weights with hardly any data flowing through them

Isn't that the idea behind sparse networks?

[ Parent | on HN ]

Re:Automatically Detecting Under-Trained Tokens in Large Lan... by Hugsun on Tuesday May 14, 2024 @02:34PM (#40358515)

I believe model distillation does this. SparseGPT was a big one, managing to remove 50% of parameters without loosing much accuracy IIRC. I saw a more recent paper citing the SparseGPT one that managed around 70-80% sparsity, pretty impressive stuff.

[ Parent | on HN ]