News for hackers. Stuff that matters.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De... by hobofan on Tuesday November 21, 2023 @02:17AM (#38360426)
Is there any data to back up the "faster to start and more resource-efficient", especially if you compare it to the native non-Python solutions that most people are using to run LLMs on local machines?

I'm as big of a fan of Rust and WASM as the next person, but throwing around claims like that without benchmarks is one of the quickest ways to get your product dismissed.

Re:Self-Hosting Open Source LLMs: Cross Devices and Local De... by 3Sophons on Tuesday November 21, 2023 @02:54AM (#38360680)
Faster is compared to Python. Portable, more secure and lightweight are compared with Python and other native solutions. In terms of benchmarks, Rust / C++ is 50,000x faster than Python; WasmEdge runtime + portable app is 30M compared with 4G Python and 300MB llama.cpp Docker image that is NOT portable across CPU or GPU; Wasm sandbox is more secure than native binary.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De... by hobofan on Tuesday November 21, 2023 @03:54AM (#38361124)
Thanks for clarifying!
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De... by 3Sophons on Tuesday November 21, 2023 @02:55AM (#38360695)