When linux was first released in 1991 a 386 to run it would cost about $2000.

We've already seen big advancements in tools to run them on lesser hardware. It wouldn't surprise me if we see some big advancements in the hardware to run them over the next few years, currently they are mostly being run of graphics processors that aren't optimised for the task.

I agree utility of open source for personal usecase is overblown.

But for commercial usecases, open source is very relevant for privacy reasons as many enterprises have strict policy not to share data with third party. Also it could be a lot cheaper for bulk inference or to have a small model for particular task.

However, the same thing could be achieved with closed source models. There's nothing to stop an LLM being made available to run on prem under a restrictive license. It would really be no different to ye olde desktop software - keeping ownership over bits shipped to a customer is solved with the law rather than technical means.

That said, I really hope open source models can succeed, it would be far better for the industry if we had a Linux of LLMs.

> Keeping ownership over bits shipped to a customer is solved with the law rather than technical means.

Yes in theory... In practice, what happened with LLaMA showed people will copy and distribute weights while ignoring the license.

Locally hosted instances that don't report on prompts is important for personal privacy.

Even a small startup, a researcher or a tinkerer can get a cloud instance with a beefy GPU. Also of note, Apple's M1 Max/Ultra should be be able to run it on their GPUs given their 64/128GB of memory, right? That's an order of magnitude cheaper.

I am confused. Those amounts are ram, not gpu ram, aren‘t they? Macs cpus are impressive, but not for ml. A most realistic one for a consumer is a 4090 rtx 24 GB. A lot of models do not fit in that, so A6000 48GB and over for some professional cards. That might be around 9000€ already.

> Macs cpus are impressive, but not for ml

On Mac GPU has access to all memory.

Apple Silicon has unified memory - all memory is accessible to both the CPU and GPU parts of the SoC.

But they comes at max 32GB model?

Mac Studio (desktop) is up to 128GB, and Macbook Pro is up to 96GB.

I overlooked the unified memory on those machines. Can it really run this performantly?

I run Vicuna quite well with my M1 Pro, 32GB.

> is it that important to open source models that can only run on hardware worth tens of thousand of dollars?

Yes, because as we've seen with other open source AI models, it's often possible for people to fork code and modify it in such a way that it runs on consumer grade hardware.

Yes, because it can always be down ported by people with more constraints than the original authors. We’ve see a lot of this in the LLM space, and a lot of other OSS efforts.

$30000 is less than price of average car that Americans buy (and most families have two of them) - that's definitely in the realm of something that affluent family can buy if it provides enough value. I also expect price to go down and at $10k it's less than mid-range bathroom update. The question is only if it provides enough value or using in the cloud better option for almost all families.

They don't only run on high end systems. Good models can run on a desktop you have at home. If you don't have a desktop... I'm not sure what you're doing on HN.

You have a weird definition of "good model"

Llama 7B is NOT a good model.

You can run much larger models than llama-7B. Galpaca-30b or Galactica-120b for example.

30B is still not good enough.

What kind of desktop are you running a 120B model on with reasonable performance?

I would disagree that 30B is not good enough. It heavily depends on which model, and what you're trying to use it for.

30B is plenty if you have a local DB of all of your files and wiki/stackechange/other important databases places in a embedding vectordb.

This is typically what is done when people make these models for their home, and it works quite well while saving a ton of money.

While llama-7B systems on their own may not be able to construct a novel ML algorithm to discover a new analytical expression via symbolic regression for many-body physics, you can still get a great linguistic interface with them to a world of data.

You're not thinking like a real software engineer here - there are a lot of great ways to use this semantic compression tool.

If I just wanted a fuzzy search engine for local data, I'd use a vector DB - there's no need for an LLM on top of that.

It will create price competition for different providers of the model though, which should drive down prices

"It only benefits bad people" is a pretty shitty argument at this point tbf. You can apply this logic to any expensive thing at this point.

I can for example, afford the hardware worth tens of thousands of dollars. I don't want to, but I can if I needed to. Does that automagically make me their competitor or a bad actor?