M2 Ultra can run 128 streams of Llama 2 7B in parallel

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by londons_explore on Wednesday October 11, 2023 @06:28PM (#37851156)

You need one good motivated sysadmin.

But if you can motivate that same sysadmin to spend his skills on something more directly benefiting your company, then you should still buy it in.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by c0pium on Wednesday October 11, 2023 @07:19PM (#37851625)

Now your business depends on that one person, congratulations. Hope they don’t realize that their skills are better suited to working at AWS/Azure etc. for 2x the money. Which they are.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by londons_explore on Wednesday October 11, 2023 @07:29PM (#37851705)

It's not core stuff. If the one sysadmin leaves, you buy in a solution.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by c0pium on Thursday October 12, 2023 @12:28AM (#37853494)

I mean, fair enough I guess. Why play the OpEx vs CapEx game when you can just pay both.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by hamandcheese on Wednesday October 11, 2023 @07:32PM (#37851734)

I think you need one motivated developer who likes to tinker with hardware for a few hours a month.

Given how popular homelabs are, I don't think this would be too hard to find.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by lghh on Wednesday October 11, 2023 @09:56PM (#37852633)

I don’t want to build a business around a dev who likes to “tinker with hardware”.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by hamandcheese on Thursday October 12, 2023 @12:56AM (#37853590)

Why the scare quotes? It's a skill like any other. Startups are built by folks who wear many hats.

[ Parent | on HN ]