Are language models deprived of electric sleep?

Re:Are language models deprived of electric sleep? by benreesman on Saturday July 23, 2022 @02:19PM (#32206001)

OPT-6.7B is good, but not even close to GPT-3.

If you can get GPT-like performance out of a 17B model, you should publish that.

[ Parent | on HN ]

Re:Are language models deprived of electric sleep? by soraki_soladead on Saturday July 23, 2022 @03:11PM (#32206581)

I’m referring to post-training pruning not smaller models. This is already well-studied but it’s not as useful as it could be on current hardware. (Deep learning currently works better with the extra parameters at training time).

Retrieval models (again, lots of published examples: RETRO, etc.) that externalize their data will bring the sizes down by about that order as well.

[ Parent | on HN ]

Re:Are language models deprived of electric sleep? by benreesman on Saturday July 23, 2022 @08:05PM (#32208851)

I agree that RETRO is cool. I think you might be stretching it a bit with the applicability, but I take your point.

[ Parent | on HN ]