Show HN: Explore large language models on any computer with 512MB of RAM

Re:Show HN: Explore large language models on any computer wi... by xrd on Saturday June 17, 2023 @12:21PM (#36371604)

At first glance i would be surprised if this works at all.

The readme says it loads a significant amount of data the first time, 250mb. None of the LLM weights I know of are less than several gigabytes in size.

It says it only requires 512mb of RAM. None of the interesting LLMs I know of run in less than 6gb of VRAM.

It says it uses no API keys, which is great, but that means inference is local, which I can't imagine works with the above constraints.

[ Parent | on HN ]

Re:Show HN: Explore large language models on any computer wi... by nattaylor on Saturday June 17, 2023 @12:34PM (#36371759)

At 512gb RAM, it uses https://huggingface.co/jncraton/LaMini-Flan-T5-248M-ct2-int8 which says

"This model is one of our LaMini-LM model series in paper "LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions". This model is a fine-tuned version of google/flan-t5-base on LaMini-instruction dataset that contains 2.58M samples for instruction fine-tuning"

[ Parent | on HN ]

Re:Show HN: Explore large language models on any computer wi... by Translationaut on Saturday June 17, 2023 @01:40PM (#36372445)

Those minified models are still equal or bigger compared to the initial "attention is all you need" transformer.

[ Parent | on HN ]