Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by 3Sophons on Tuesday November 21, 2023 @02:09AM
(#38360381)
While Mistral's Docker image provides a practical solution, it's important to note that Docker isn't truly cross-platform. A Docker image compiled for x86 architecture won't run on ARM, which can be a limitation for diverse hardware environments. On the other hand, Rust + WebAssembly (Wasm) offers genuine cross-platform capabilities. Wasm is not only faster to start and more resource-efficient but also more secure. This makes it exceptionally suitable for handling large language models, ensuring high performance while being cost-effective. In scenarios where diverse hardware support and efficiency are crucial, Rust + Wasm emerges as a superior choice over Docker.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by hobofan on Tuesday November 21, 2023 @02:17AM
(#38360426)
Is there any data to back up the "faster to start and more resource-efficient", especially if you compare it to the native non-Python solutions that most people are using to run LLMs on local machines?
I'm as big of a fan of Rust and WASM as the next person, but throwing around claims like that without benchmarks is one of the quickest ways to get your product dismissed.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by 3Sophons on Tuesday November 21, 2023 @02:54AM
(#38360680)
Faster is compared to Python. Portable, more secure and lightweight are compared with Python and other native solutions. In terms of benchmarks, Rust / C++ is 50,000x faster than Python; WasmEdge runtime + portable app is 30M compared with 4G Python and 300MB llama.cpp Docker image that is NOT portable across CPU or GPU; Wasm sandbox is more secure than native binary.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by smcleod on Tuesday November 21, 2023 @06:29AM
(#38362245)
Usually when you’re building container images at any scale you’d use something like buildx, kaniko or buildah which allow you to easily set multiple target architectures (AMD64, ARM64) for your images.
There’s hardly any / basically no overhead to running an application in a container if that’s what you mean by overhead. If you mean the image size - well you only need to add the things you need - the problem is people tend to abuse images and install a lot of packages in the final image which absolutely aren’t required.
Rust or golang at the core would be nice as Python can be slow at times. I do wish more folks would give Tauri a go for GUI apps.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by rgbrgb on Tuesday November 21, 2023 @09:42AM
(#38364254)
I get similar speeds on my m1, though it depends a bit on the quantization. If anyone else is on Mac and just want to have a play, FreeChat (my app) is a simple chat wrapper for llama.cpp. Clicking the little … on a message and checking the info will tell you the generation speed.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by jesterson on Tuesday November 21, 2023 @02:03AM
(#38360330)
Does anyone have any feedback on using those open-source models with any language except English? Particularly non-western group of languages like korean/japanese/chinese?
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by numpad0 on Tuesday November 21, 2023 @04:27AM
(#38361394)
It's probably better to switch to suitable language-specific models. GPTs sound translated in non-native languages, and also has difficulty separating CJK due to that Unicode quirk.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by axegon_ on Tuesday November 21, 2023 @04:40AM
(#38361486)
A bit of a hit and miss really. I haven't fiddled with languages all that much, but I was kinda curious to try a bunch of models with Spanish(the only large language other than English in my toolbox) about a month or two ago. Aguila 7B was the one that really stood out but then again it shouldn't come as a surprise - it is practically Falcon 7B trained on Spanish data.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by 3Sophons on Tuesday November 21, 2023 @04:21AM
(#38361339)
Ollama's approach with Docker is indeed user-friendly, but Rust + WebAssembly (Wasm) offers some cool perks that are worth considering. With Wasm, you get true cross-platform support, meaning it runs smoothly on various devices, automagically uses the local hardware accelerator to run at full speed. It's also super lightweight, making it faster and more resource-efficient – a big plus for running LLMs, especially on devices with limited resources. A Wasm inference application can be as small as 2MB, including all dependencies, which is significantly smaller than most Docker containers.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by hobofan on Tuesday November 21, 2023 @04:39AM
(#38361477)
Ollama doesn't use Docker by default. While it is one of the install/download options, the default mode of operation is installing it as a native application (where they provide prebuilt binaries for the main platforms/architectures) into which you can easily load models from HuggingFace repos or local GGUF files.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by hiatus on Tuesday November 21, 2023 @10:11AM
(#38364663)
I don't understand. When you run `ollama pull` (or even `ollama run` which will pull if you don't have the model), that is using docker is it not? The code even references pulling from a registry.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by hobofan on Tuesday November 21, 2023 @10:35AM
(#38365030)
I haven't looked to deeply into the code, but it looks to me like it just uses OCI/Docker images as a distribution format.
You can find some evidence of that in the model directory (mentioned in the FAQ[0]), where you can find manifest files that have contents of a docker manifest.
Using this format likely makes distribution a lot easier as they can just use an existing OCI registry for hosting the images, and you can use existing push/pull primitives. OCI images are often used nowadays for things besides docker images if you have a format (like a layered filesystem) that is nicely content-addressable.
The default registry for Ollama is https://ollama.ai/library (I don't know about the API endpoint). I'm not sure if/how you can actually use it with another registry endpoint.
When people say that something "uses Docker" that usually involves utilizing the Docker daemon with it's sandboxing, networking, etc.. Ollama doesn't do or need to do that as it can just do inference natively based on GGUF files (+ the other configuration that comes as part of their model file).
However, as you have noticed, it does borrow heavily from Docker on a conceptual level. E.g. apart from the registry mechanism its Modelfile heavily mimics a Dockerfile.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by jsight on Tuesday November 21, 2023 @10:34AM
(#38365005)
ollama maintains a registry of ollama compatible llms. I don't see any indication that they use docker for that registry, but I could have overlooked something. I also don't see any docker/podman containers running when using ollama.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by LoganDark on Tuesday November 21, 2023 @07:06AM
(#38362534)
> With Wasm, you get true cross-platform support, meaning it runs smoothly on various devices, automagically uses the local hardware accelerator to run at full speed.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by Kubuxu on Tuesday November 21, 2023 @09:05AM
(#38363731)
> you get true cross-platform support
wasi_nn is just an abstraction layers for wrappers around llama.cpp/gguf.cpp/onyx/others. It is not any more cross-platform than these libraries/apps deployed themselves, or rust apps using bindings for them.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by 3Sophons on Tuesday November 21, 2023 @09:34PM
(#38373931)
The demos are as fast as the ChatGPT, they just look slower. -First. It might seem slow at first because it's loading the model. But once that's done, it gets much quicker for any follow-up requests. Second, Turning on streaming output helps a lot, as it shows responses as they're processed. And yes, the hardware matters too. A good GPU setup in the cloud can work wonders, though it might bump up the cost a bit. So actually the demo's not slower than ChatGPT overall; it just takes a moment to warm up at the start.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by MaKey on Tuesday November 21, 2023 @06:19AM
(#38362163)
What are Mistrals strengths and weaknesses? I tried it for infrastructure as code and it wasn't able to output more than the most basic examples, let alone modify them.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by hobofan on Tuesday November 21, 2023 @06:46AM
(#38362370)
Mistrals main strength is that it is a good allrounder model with a 7B size, so it runs on a lot of consumer hardware. If you are looking for specific tasks to run locally (with decent RAM and no GPU), the realistic best option in most categories is a fine-tuned 13B model.
In my experience most open source models do quite well with Python and Javascript, but don't perform so well when it comes to other languages and the usage of their special characteristics (e.g. it will still be able to write simple control flow in Rust, but probably not use async/await correctly).
I've tried using LLMs in general with IaC (K8s resources/Helm charts), and they all did well when asking it a very specific thing about it (e.g. "Is there an alternative way to accomplish X?"), but also never had it perform well when outputting YAML directly (either from scratch of modifying it).
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by smcleod on Tuesday November 21, 2023 @07:17AM
(#38362627)
I assume you mean IaC? It’s how you manage infrastructure with automation. Terraform, Ansible, CDK etc… back in the day it used to be for managing pets with Pupper, Chef and friends.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by youngNed on Tuesday November 21, 2023 @10:54AM
(#38365351)
Ashu, i am going to guess that english is not your first language, apologies, if that is not the case, but " Tell me how can I run it locally." is not a great way to ask for info, not here at least. A 'please' would certainly help, but a less direct way might be even better.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by jasonjmcghee on Tuesday November 21, 2023 @11:47AM
(#38366153)
Which did you try? Many of these models don't work well at all for queries like "please do this task for me". But then people fine tune them and they work much better.
Give zephyr a try (available in ollama and similar places)
It's a fine tune of mistral and works quite well.
But as you point out, these models have less general knowledge compared to their massive siblings. Knowledge based queries are going to be lower quality than using gpt-4.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by tormeh on Tuesday November 21, 2023 @10:25AM
(#38364847)
The WasmEdge README gives me the heebie-jeebies. Starry-eyed emojis, highlights use-case for today's most trendy thing even though it's general-purpose, mentions blockchain. This reeks of former cryptobros chasing the next big thing. I'd trust Wasmtime more.
https://docs.mistral.ai/quickstart/
I'm as big of a fan of Rust and WASM as the next person, but throwing around claims like that without benchmarks is one of the quickest ways to get your product dismissed.
There’s hardly any / basically no overhead to running an application in a container if that’s what you mean by overhead. If you mean the image size - well you only need to add the things you need - the problem is people tend to abuse images and install a lot of packages in the final image which absolutely aren’t required.
Rust or golang at the core would be nice as Python can be slow at times. I do wish more folks would give Tauri a go for GUI apps.
[0]: https://www.freechat.run/
Will assess myself but wonder if anyone tried.
and also Baichuan https://www.secondstate.io/articles/baichuan2-13b-chat/
There is an Arabic one called Jais mentioned by Satya a few days ago on the Microsoft dev day would like to try out
I read somewhere that Mistral 7B had a similar performance to GPT-3, but it seems to be miles behind it unfortunately.
To be fair to the developers of Mistral, it's still amazing that I can run this on my computer, and it's certainly better than the first LLaMA.
https://github.com/jmorganca/ollama/blob/main/cmd/cmd.go#L95...
The default registry for Ollama is https://ollama.ai/library (I don't know about the API endpoint). I'm not sure if/how you can actually use it with another registry endpoint.
When people say that something "uses Docker" that usually involves utilizing the Docker daemon with it's sandboxing, networking, etc.. Ollama doesn't do or need to do that as it can just do inference natively based on GGUF files (+ the other configuration that comes as part of their model file).
However, as you have noticed, it does borrow heavily from Docker on a conceptual level. E.g. apart from the registry mechanism its Modelfile heavily mimics a Dockerfile.
[0]: https://github.com/jmorganca/ollama/blob/aabd71aede99443d585...
No... no, that is not what Wasm means at all.
wasi_nn is just an abstraction layers for wrappers around llama.cpp/gguf.cpp/onyx/others. It is not any more cross-platform than these libraries/apps deployed themselves, or rust apps using bindings for them.
In my experience most open source models do quite well with Python and Javascript, but don't perform so well when it comes to other languages and the usage of their special characteristics (e.g. it will still be able to write simple control flow in Rust, but probably not use async/await correctly).
I've tried using LLMs in general with IaC (K8s resources/Helm charts), and they all did well when asking it a very specific thing about it (e.g. "Is there an alternative way to accomplish X?"), but also never had it perform well when outputting YAML directly (either from scratch of modifying it).
Pretty good at instructions
Uncensored, no RHLF bs
Anyway, to answer your question: https://ollama.ai/
Give zephyr a try (available in ollama and similar places)
It's a fine tune of mistral and works quite well.
But as you point out, these models have less general knowledge compared to their massive siblings. Knowledge based queries are going to be lower quality than using gpt-4.