News for hackers. Stuff that matters.

A comment on Llama2.c L2E LLM – Multi OS Binary and Unikernel Release.

Re:Llama2.c L2E LLM – Multi OS Binary and Unikernel Release by ragnarok123 on Saturday August 26, 2023 @04:52AM (#37270912)
Does it offer a local api so i can embed this in my python build?
[ Parent | on HN ]
Re:Llama2.c L2E LLM – Multi OS Binary and Unikernel Release by AMICABoard on Saturday August 26, 2023 @07:29AM (#37271795)
not yet
[ Parent | on HN ]
Re:Llama2.c L2E LLM – Multi OS Binary and Unikernel Release by sa-code on Saturday August 26, 2023 @08:10AM (#37272082)
The ability to hook it into frameworks like streamlit would be huge. Sorta like llama-cpp-python
[ Parent | on HN ]
Re:Llama2.c L2E LLM – Multi OS Binary and Unikernel Release by AMICABoard on Saturday August 26, 2023 @08:16AM (#37272124)
Agreed and yes would be awesome... But currently this does not infer large meta 7b model efficiently - like 1 token or lower per seconds. But the small toy story (not so useful) models are fast.

If the above mentioned API / python binding is ready, I'll make a streamlit interface demo. The streamlit demo should be simple. But I have to figure out python binding.

[ Parent | on HN ]
Re:Llama2.c L2E LLM – Multi OS Binary and Unikernel Release by AMICABoard on Saturday August 26, 2023 @08:13AM (#37272107)
If you add an issue you can do so on https://github.com/trholding/llama2.c

I have a planned a web api. Are you looking for python binding? I'm interested to know. The issues would keep me organized.

[ Parent | on HN ]