M2 Ultra can run 128 streams of Llama 2 7B in parallel

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by c0pium on Wednesday October 11, 2023 @07:19PM (#37851625)

Now your business depends on that one person, congratulations. Hope they don’t realize that their skills are better suited to working at AWS/Azure etc. for 2x the money. Which they are.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by londons_explore on Wednesday October 11, 2023 @07:29PM (#37851705)

It's not core stuff. If the one sysadmin leaves, you buy in a solution.

[ Parent | on HN ]

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by c0pium on Thursday October 12, 2023 @12:28AM (#37853494)

I mean, fair enough I guess. Why play the OpEx vs CapEx game when you can just pay both.

[ Parent | on HN ]