Re:GPT-4 Details Leaked
by moffkalast on Tuesday July 11, 2023 @04:59AM
(#36678229)
Idk, has anyone tried falcon yet? The support for running it remains nonexistent except for one fork of llama.cpp that isn't integrated into anything. This trend of every new model breaking compatibility really needs to stop.
[ Parent
| on HN ]
I have and I am running it locally. Mostly in 7b variant which runs pretty much at chatgpt speed (when streaming) on my ryzen 3700 cpu + 32gb ram + 2xnvme in a mirror (it doesn't fit in the memory in its entirety, few gb go into the swap).
Of course to run it like that I have to be running nothing else. No xorg, no chromium etc. Just a pure linux console.
If you want to try falcon 40b instruct by yourself here I'd a public demo : https://huggingface.co/blog/falcon
Go to the bottom of the page.