I think it’s the opposite of woke, it’s slept!
3-4gb vram or cpu (1.3b): https://huggingface.co/TheBloke/deepseek-coder-1.3b-instruct...
Alternative for chat (1.3b): https://huggingface.co/TheBloke/evolvedSeeker_1_3-GGUF
Alternative for chat (3b): https://huggingface.co/TheBloke/open-llama-3b-v2-wizard-evol...
6-8gb vram (6.7b): https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct...
Self hosted 6.7b model is phenomenal in my experience, I actually prefer it to chatgpt a lot of the time. Similar quality code but fewer disclaimers and #todo in my experience. Gpt4 is still a bit better at coding but not by much and it's much slower for me.
This can also significantly reduce its bias since you are in control of the system prompt. But also, even ChatGPT can be trivially made to behave differently by saying that you're writing a book or making a video game etc, describing a character in it, and then asking it how that character would have responded in such and such situation.
1 = ```
$ xcode-select --install
xcode-select: error: command line tools are already installed, use "Software Update" in System Settings to install updates
```
$ sha256sum < llamafile-server-0.1-llava-v1.5-7b-q4
a138c5db9cff3b8905dd6e579c2ab6c098048526b53ae5ab433ff1d1edb9de24 -
$ ./llamafile-server-0.1-llava-v1.5-7b-q4
Killed: 9 $ ./llamafile-server-0.1-llava-v1.5-7b-q4
[2] 25224 illegal hardware instruction ./llamafile-server-0.1-llava-v1.5-7b-q4from https://dortania.github.io/OpenCore-Legacy-Patcher/MODELS.ht...
I thought my 2015 MBP wasn't able to upgrade. Good to know it's still supported.
All the new AI toys especially seem to love beefy newish hardware and especially GPU hardware if available
$ chmod +x llamafile-server-0.1-llava-v1.5-7b-q4
$ ./llamafile-server-0.1-llava-v1.5-7b-q4
run-detectors: unable to find an interpreter for ./llamafile-server-0.1-llava-v1.5-7b-q4
Hmm. Did I do something wrong? (Ubuntu 22.04 / )Installing the portable binfmt_misc gets me further, but still:
$ ./llamafile-server-0.1-llava-v1.5-7b-q4
zsh: permission denied: ./llamafile-server-0.1-llava-v1.5-7b-q4
$ sh -c ./llamafile-server-0.1-llava-v1.5-7b-q4
sh: 1: ./llamafile-server-0.1-llava-v1.5-7b-q4: Permission denied sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf
sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
You can solve the zsh permission denied issue by either (1) upgrade to zsh 5.9+ (I upstreamed a fix for this bug in zsh two years ago) or (2) use the sh -c workaround you discovered. If that one doesn't work, then it likely needs to be chmod +x. If the execute bit is set, and your sh still isn't working, then please let me know, because I'm not aware of any sh that still doesn't support APE.See the Gotchas section of the README https://github.com/mozilla-Ocho/llamafile#gotchas
(sorry for OT!)
EDIT: oh, wait. Actually people usually have a handful to a few dozen of the these models lying around. When they update their backend, you just redownload every single model again.
EDIT 2: right, you can release a program that automatically patches and updates the downloaded model+executables. Such an invention.
unzip -l llamafile-server-0.1-llava-v1.5-7b-q4 | grep llava-v1
Archive: llamafile-server-0.1-llava-v1.5-7b-q4
4081004224 11-15-2023 22:13 llava-v1.5-7b-Q4_K.gguf
177415936 11-15-2023 22:13 llava-v1.5-7b-mmproj-Q4_0.ggufBut yes, unless there is a way to patch it, bundling the model with the executable like this is going to be more wasteful.
> llamafile-server-0.1 -m llama-2-13b.Q8_0.gguf
You know, most people don't have 24+GB GPUs sitting around to train these models. So in my book this is a huge step forward. Personally, this is the first time i am able to run an LLM on my computer, and it's purely thanks to this.
In case anyone here hasn't used chatgpt-web, I recommend trying it out. With the new GPT-4 models you can chat for way cheaper than paying for ChatGPT Plus, and you can also switch back to the older (non-nerfed) GPT-4 models that can still actually code.
For GPT-4 the pricing is double that (0.09 USD per 1K). So only 200 larger interactions to reach 20 USD cost.
Or am I wrong?
Now, if the chat gets very long or is heavy on high-token strings (especially code), those costs can balloon up to the 9-12 cent region. I think this is because chatgpt-web loads all the prior messages in the chat into the context window, so if you create a new chat for each question you can lower costs substantially. Most often I don't need much prior context in my chat questions anyway, as I use ChatGPT more like StackOverflow than a conversation buddy.
Also, it's a lot easier to run company subscriptions this way, as we don't have to provision a new card for each person to use the web version. I believe there is an Enterprise version of ChatGPT, but chatgpt-web is functionally equivalent and I'm sure it costs less.
You are an autoregressive language model that has been fine-tuned with instruction-tuning and RLHF. You carefully provide accurate, factual, thoughtful, nuanced answers, and are brilliant at reasoning. If you think there might not be a correct answer, you say so.
Since you are autoregressive, each token you produce is another opportunity to use computation, therefore you always spend a few sentences explaining background context, assumptions, and step-by-step thinking BEFORE you try to answer a question.
Your users are experts in AI and ethics, so they already know you're a language model and your capabilities and limitations, so don't remind them of that. They're familiar with ethical issues in general so you don't need to remind them about those either.
Don't be verbose in your answers, keep them short, but do provide details and examples where it might help the explanation. When showing code, minimize vertical space.
I'm hesitant to share it because it works so well, and I don't want OpenAI to cripple it. But, for the HN crowd...Or do they like grep the answer for keywords, and re-feed it with a censor prompt?
Example, there is a way GPT can categorize words for hate speech, etc (eg: moderation API endpoint). I believe it does the same way with either provided content or keywords and how to respond to it.
Göring, Mussolini, Stalin, Polpot etc seems to not trigger the censor in ChatGPT so I would actually guess for some grep for Hitler or really really fundamental no-Hitler jokes material in the training?
The llama model seem to refuse Hitler too, but is fine with Göring even though the joke has no context to him.
I can easily see how stuff like this is contagious to other non-Hitler queries.
If ChatGPT is using this model then it's more reasonable to assume that they are bleeding money and need to cut costs.
People really need to stop asking ChatGPT to write out complete programs in a single prompt.
I'd expect we see improved behavior in the coming weeks.
https://github.com/tluyben/llamafile-docker
to save even more keystrokes.
Looks like cargo cult for me.
Cargo cult is not in using Docker, but in using Docker to wrap something already wrapped into a comparable layer of abstraction.
Besides,
> not polluting my OS (filesystem and processes) with bits and bobs I downloaded off the internet
is purely self-deception. It's not like Docker images are not stored in some folder deep in the filesystem. If anything, it's harder to clean up after Docker than just doing rm -rf on a directory with llamafiles.
There are tradeoffs hers. For some people the tradeoff of a single executable file with everything in it compared to setting up a Docker system is worthwhile.
I am not trusting that a tradition virus / etc scanner will find stuff hidden in executable models.
> User: What is the third planet from the sun?
> Llama: The third planet from the sun is called Mars.
Ask it if is there life on Mars in that parallel reality
Also, the knowledge can be kind of siloed. You often have to come at it in weird ways. Also, they are not fact-bases. They are next-token-predictors, with extra stuff on top. So if people on the internet often get the answer wrong, so will the model.
Can't wait to unleash Pluto questions.
EDIT: never mind, other commenters here answered my question: yes it does work. I have a M2Pro with 32 G on chip memory - amazingly effective for experimenting with open LLMs.
This seems like a great approach to compare multiple models, in particular.
1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/...:
2. Make that binary executable, by running this in a terminal: 3. Run your new executable, which will start a web server on port 8080: 4. Navigate to http://127.0.0.1:8080/ to upload an image and start chatting with the model about it in your browser.Screenshot here: https://simonwillison.net/2023/Nov/29/llamafile/