MLC-LLM: GPT/Llama on consumer-class GPUs and phones

Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by valine on Sunday April 30, 2023 @02:51PM (#35765448)

Sure, but other than Apple no one has the tight integration needed to make that happen. If Apple leans into their CoreML stuff they have a real opportunity to steal the market from Nvidia.

Long term maybe Nvidia is able to release release some integrated ARM chip, but I’m not holding my breath.

[ Parent | on HN ]

Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by verdverm on Sunday April 30, 2023 @03:08PM (#35765634)

Nvidia sells data center products, Apple is not going to steal this market, they aren't even in it. The gaming market is dominated by Windows, I don't see this moving meaningfully just because apple has some marginally better hardware at this point in time.

The Risc V area, now there we can talk about disruption long term

[ Parent | on HN ]

Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by amelius on Sunday April 30, 2023 @03:44PM (#35765957)

NVidia Jetson is a line of products with tight integration between CPU and GPU. As just one example.

[ Parent | on HN ]

Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by nl on Sunday April 30, 2023 @08:12PM (#35767757)

Does it? I'm not aware of any way to share main memory with the Jetson CUDA cores.

If I'm wrong I'd love a pointer to the docs about it!

[ Parent | on HN ]

Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by dhruvdh on Sunday April 30, 2023 @05:46PM (#35766895)

How do you imagine consoles work? And what about Apple's tight integration has actually made itself useful? The developer experience is terrible.

[ Parent | on HN ]

Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by valine on Sunday April 30, 2023 @06:07PM (#35767028)

It’s useful because you can buy a Macbook with 64GB of RAM, and then use that RAM as VRAM to run your LLM.

[ Parent | on HN ]