Running a 180B parameter LLM on a single Apple M2 Ultra

Re:Running a 180B parameter LLM on a single Apple M2 Ultra by bugglebeetle on Thursday September 07, 2023 @12:38PM (#37421627)

I think it’s the opposite. Models will become more commoditized and closed/invisible as the basis of other service offerings. Apple isn’t going to start offering general API access to the model they’re training, but will bake it into a bunch of stuff and maybe give platform developers limited access. Meta will probably continue to drive the commoditization train because they have a killer ML/AI team, but the same thing will likely happen there once it’s the basis for a service that generates money.

[ Parent | on HN ]

Re:Running a 180B parameter LLM on a single Apple M2 Ultra by foobiekr on Thursday September 07, 2023 @01:17PM (#37422273)

This. We haven’t even entered the get-serious monetization era.

Now that the infinite free money pump has been turned down a bunch, we’re going to see what reality looks like.

[ Parent | on HN ]

Re:Running a 180B parameter LLM on a single Apple M2 Ultra by 6510 on Friday September 08, 2023 @02:28AM (#37430155)

Okay ill tell you. You need to start a startup that sets up a good number of cameras at manual labor jobs. Most of the footage will be completely useless but every day you hit a once in a day event, every week you get a once in a week event, every month, every year, every decade etc! Then the guy working there for 40 years wacks pipe 224 with a hammer 50 cm from the outlet and production resumes.

The footage can be aggressively pruned to fit on the disk.

When the robot is delivered in 2033 it can easily figure out, from the footage, all these weird and rare edge cases.

The difference will be like that between a competent but new employee and someone with 10 years of experience.

I can see the Tesla bots disassembling the production line already. Or do you think it wont happen?

[ Parent | on HN ]

Re:Running a 180B parameter LLM on a single Apple M2 Ultra by Aerbil313 on Sunday September 10, 2023 @07:14AM (#37454779)

Transformers can and do forget.

[ Parent | on HN ]

Re:Running a 180B parameter LLM on a single Apple M2 Ultra by checkyoursudo on Wednesday September 13, 2023 @08:05AM (#37495611)

Assuming that this would work, which I am fine with granting for purposes of discussion, how does this method ever let you build anything new? Or make use of advances in production methods? Or completely reconfigure a production line because of some regulatory requirement?

[ Parent | on HN ]

Re:Running a 180B parameter LLM on a single Apple M2 Ultra by cs702 on Thursday September 07, 2023 @01:54PM (#37422892)

Actually, we're saying the same thing: Models are becoming more commoditized, so profits will accrue, not to those companies who say they have the "best" models, but to the companies that have other kinds of advantages.

[ Parent | on HN ]