News for hackers. Stuff that matters.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De... by politelemon on Tuesday November 21, 2023 @01:17AM (#38360044)
Mistral have created a docker image which hosts their model in vllm. Vllm creates an openai like http API interface.

https://docs.mistral.ai/quickstart/

Re:Self-Hosting Open Source LLMs: Cross Devices and Local De... by 3Sophons on Tuesday November 21, 2023 @02:09AM (#38360381)
While Mistral's Docker image provides a practical solution, it's important to note that Docker isn't truly cross-platform. A Docker image compiled for x86 architecture won't run on ARM, which can be a limitation for diverse hardware environments. On the other hand, Rust + WebAssembly (Wasm) offers genuine cross-platform capabilities. Wasm is not only faster to start and more resource-efficient but also more secure. This makes it exceptionally suitable for handling large language models, ensuring high performance while being cost-effective. In scenarios where diverse hardware support and efficiency are crucial, Rust + Wasm emerges as a superior choice over Docker.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De... by hobofan on Tuesday November 21, 2023 @02:17AM (#38360426)
Is there any data to back up the "faster to start and more resource-efficient", especially if you compare it to the native non-Python solutions that most people are using to run LLMs on local machines?

I'm as big of a fan of Rust and WASM as the next person, but throwing around claims like that without benchmarks is one of the quickest ways to get your product dismissed.

Re:Self-Hosting Open Source LLMs: Cross Devices and Local De... by 3Sophons on Tuesday November 21, 2023 @02:54AM (#38360680)
Faster is compared to Python. Portable, more secure and lightweight are compared with Python and other native solutions. In terms of benchmarks, Rust / C++ is 50,000x faster than Python; WasmEdge runtime + portable app is 30M compared with 4G Python and 300MB llama.cpp Docker image that is NOT portable across CPU or GPU; Wasm sandbox is more secure than native binary.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De... by hobofan on Tuesday November 21, 2023 @03:54AM (#38361124)
Thanks for clarifying!
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De... by 3Sophons on Tuesday November 21, 2023 @02:55AM (#38360695)
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De... by smcleod on Tuesday November 21, 2023 @06:29AM (#38362245)
Usually when you’re building container images at any scale you’d use something like buildx, kaniko or buildah which allow you to easily set multiple target architectures (AMD64, ARM64) for your images.

There’s hardly any / basically no overhead to running an application in a container if that’s what you mean by overhead. If you mean the image size - well you only need to add the things you need - the problem is people tend to abuse images and install a lot of packages in the final image which absolutely aren’t required.

Rust or golang at the core would be nice as Python can be slow at times. I do wish more folks would give Tauri a go for GUI apps.