News for hackers. Stuff that matters.
Re:OpenAI's plans according to Sam Altman by usrbinbash on Tuesday June 06, 2023 @04:39AM (#36209909)
> Well, the entire idea is to run locally, right? Now you have to worry about getting the runtime to work on 3 different OSs,

You mean, like a runtime that can be a single, statically linked binary, that requires a c-compiler? Which already has bindings for several popular languages? https://github.com/ggerganov/llama.cpp/

Yeah, maybe I am an optimist, but somehow I don't see that being a big problem...

> I expect LLMs to be a ubiquitous UI paradigm.

I don't expect that to be the case. LLMs are useful in text wrangling tasks, for which we have specialized software. Software that already bundles large amounts of libraries.

And btw. you know what I also expect from a UI? That it is fully functional even when the machine is offline, and that it's responsiveness is independent from the providers current traffic load.

> But non-cloud LLMs are expected to run locally on clients.

They don't have to. I can already see many useful applications for LLMs that run in on-premises servers. I never said that open source LLMs have to be restricted to client-devices.

> All that said, perhaps we should look at legislation as well as the possibility of a OS provider playing nice.

Legislation is another good argument for open source LLMs taking over. Looking at the EU AI Act, and the recent ideas coming up in the US, it looks like model training data being open for audits is gonns be a thing in the future, especially if the model is used in a context a lot of peoples livelihoods could rely on.

Re:OpenAI's plans according to Sam Altman by yyyk on Tuesday June 06, 2023 @06:47AM (#36210761)
>>> Well, the entire idea is to run locally, right? Now you have to worry about getting the runtime to work on 3 different OSs,

>llama.cpp

I'm well aware of llama.cpp, and run it myself. I'm also aware of the recent breaking change to the quantization format because 'everyone can just rebuild from the ggml'. And its difficulties with base level AVX or AVX512 (in fairness it focuses on M2 and has support for AVX2). And how you need to work to add additional non-Llama models to it (necessary due to the commercial clause). It's not quite ready yet. Even when it's ready, there's the work for actual deployment, and the mess of diverse user systems beyond what devs tend to run.

I'm not saying the challenge is impossible - but it's an extra hurdle for OSS.

>> I expect LLMs to be a ubiquitous UI paradigm.

> I don't expect that to be the case. LLMs are useful in text wrangling tasks, for which we have specialized software

Look at the recent photoshop generative fill feature for what could be done.

>what I do expect from an UI: a) that it is fully functional even when the machine is offline, and b) that it's responsiveness is independent from the load on someones datacenter.

Good ideas. I think for now BigCorp can avoid it by pushing LLMs as an addon UI and keeping the base UI functional. It's possible they'd ship smaller models on device too.

>Legislation is another good argument for open source LLMs taking over.

That I can buy. The legal details are however a bit beyond me right now, and I don't know what the current status is. It could push OSS or stop it.