News for hackers. Stuff that matters.

A comment on NanoGPT. Parent comment

Re:NanoGPT by make3 on Wednesday January 11, 2023 @04:56AM (#34336820)
small players will never have a chance to push the state of the art, as whatever optimization there is will also be applied at large scale with more money
[ Parent | on HN ]
Re:NanoGPT by hankman86 on Wednesday January 11, 2023 @05:16AM (#34336917)
Take a leaf from Seti@Home‘s book and try to come up with a distributed, volunteer-based approach to training an open source LLM. There is already an enormous amount of suitable ML hardware on end user devices.
[ Parent | on HN ]
Re:NanoGPT by Der_Einzige on Wednesday January 11, 2023 @06:00AM (#34337153)
Huggingface actually recently did this, but I think it's for inference on their giant models like BLOOM
[ Parent | on HN ]
Re:NanoGPT by cypress66 on Wednesday January 11, 2023 @05:01AM (#34336843)
A lot of SOTA comes from small players. It just isn't the case for LLMs.
[ Parent | on HN ]
Re:NanoGPT by awestroke on Wednesday January 11, 2023 @05:02AM (#34336849)
Good point, but perhaps a leap could take small players into territories of language models that are large enough to be useful. GPT-3 crossed that threshold
[ Parent | on HN ]