LLaMA-7B in Pure C++ with full Apple Silicon support

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by xiphias2 on Friday March 10, 2023 @03:43PM (#35100726)

George Hotz already implemented LLaMA 7B and 15B on Twitch yesterday on GPU in Tunygrad llama branch:

https://github.com/geohot/tinygrad/tree/llama

The only problem is that it's swapping on 16GB Macbook, so you need at least 24GB in practice.

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by attractivechaos on Friday March 10, 2023 @04:03PM (#35101021)

Both are very impressive. A nice thing about Gerganov's implementation is that it is written in almost pure C. Arguably easier for deployment.

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by AlecSchueler on Friday March 10, 2023 @06:31PM (#35102817)

What does almost mean in this case?

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by bigstumpy on Friday March 10, 2023 @07:12PM (#35103267)

There is also C++

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by codetrotter on Friday March 10, 2023 @07:33PM (#35103488)

Iow it probably wouldn’t compile with an actual C only compiler, but by and large it looks more like C than like C++?

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by apetresc on Friday March 10, 2023 @03:48PM (#35100802)

Can you link to the Twitch stream VoD?

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by vopi on Friday March 10, 2023 @04:00PM (#35100970)

https://www.twitch.tv/georgehotz/videos?filter=archives&sort...

although, there is a VOD channel on YT that might be better.

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by yieldcrv on Friday March 10, 2023 @04:02PM (#35100992)

why does it say video unavailable?

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by vopi on Friday March 10, 2023 @04:05PM (#35101048)

Hmm. I'm not sure. Presumably this will be updated soon: https://www.youtube.com/@geohotarchive/videos

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by stagger87 on Friday March 10, 2023 @04:05PM (#35101063)

I'm pretty sure he only makes his past twitch streams available to subscribers. It's weird they are available on Youtube, maybe he doesn't know that's something he can change.

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by jonas21 on Friday March 10, 2023 @05:16PM (#35101973)

I don't think he runs the YouTube channel - it's managed by some of his fans. But like what's he gonna do about it? Send them a takedown notice?

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by botdan on Friday March 10, 2023 @06:00PM (#35102473)

Not sure about his more recent content but he used to have a policy that the VODs could be uploaded in full as long as they were uncut and not modified. If you go watch some older VODs he says so at the beginning.

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by Mistletoe on Friday March 10, 2023 @04:22PM (#35101264)

This is such a refreshing and neat way to use Twitch.

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by jalino23 on Friday March 10, 2023 @03:45PM (#35100765)

that is so impressive!

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by BoberMod on Friday March 10, 2023 @06:10PM (#35102586)

There is also a gpu-acelerated fork of the original repo

https://github.com/remixer-dec/llama-mps

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by xiphias2 on Friday March 10, 2023 @10:03PM (#35104756)

> For 7B model, it always goes above 32gb of RAM,

That's double of what Tinygrad uses

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by DennisAleynikov on Sunday March 12, 2023 @01:03AM (#35117224)

Tinygrad is using openCL right?

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by lostmsu on Friday March 10, 2023 @06:55PM (#35103088)

It's not tinygrad really, it is PyTorch.

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by capableweb on Friday March 10, 2023 @08:03PM (#35103815)

Unfamiliar with the domain, but is it? Looking through the changes (https://github.com/geohot/tinygrad/compare/master...llama) it seems to not mention pytorch at all but tinygrad + numpy.

[ Parent | on HN ]

Re:LLaMA-7B in Pure C++ with full Apple Silicon support by diimdeep on Saturday March 11, 2023 @01:39AM (#35106069)

There are two coding sessions streamed so far, very interesting to watch

George Hotz | Programming | can we fit a LLaMA inside a tinygrad? https://www.youtube.com/watch?v=0kRDs9BW2NU

George Hotz | Programming | ChatLLaMA: get in losers we're building a chatbot https://www.youtube.com/watch?v=nctqc8FBJ2U

[ Parent | on HN ]