News for hackers. Stuff that matters.

A comment on Llama2.c L2E LLM – Multi OS Binary and Unikernel Release. Parent comment

Re:Llama2.c L2E LLM – Multi OS Binary and Unikernel Release by sa-code on Saturday August 26, 2023 @08:10AM (#37272082)
The ability to hook it into frameworks like streamlit would be huge. Sorta like llama-cpp-python
[ Parent | on HN ]
Re:Llama2.c L2E LLM – Multi OS Binary and Unikernel Release by AMICABoard on Saturday August 26, 2023 @08:16AM (#37272124)
Agreed and yes would be awesome... But currently this does not infer large meta 7b model efficiently - like 1 token or lower per seconds. But the small toy story (not so useful) models are fast.

If the above mentioned API / python binding is ready, I'll make a streamlit interface demo. The streamlit demo should be simple. But I have to figure out python binding.

[ Parent | on HN ]