News for hackers. Stuff that matters.

A comment on Llama.cpp AI Performance with the GeForce RTX 5090 Review.

Re:Llama.cpp AI Performance with the GeForce RTX 5090 Review by KronisLV on Monday March 10, 2025 @07:20AM (#43319235)
Sometimes I daydream about a world where GPUs just have the equivalent of LPCAMM and you could put in as much RAM as you can afford and as much as the hardware supports, much like is the case with motherboards, even if something along the way would bottleneck somewhat. It'd really extend the life of some hardware, yet companies don't want that.

That said, it's cool that you can even get an L4 with 24 GB of VRAM that actually performs okay, yet is passively cooled and consumes like 70W, at that point you can throw a bunch of them into a chassis and if you haven't bankrupted yourself by then, they're pretty good.

I did try them out on Scaleway, the pricing isn't even that exorbitant, using consumer GPUs for LLM use cases doesn't quite hit the same since.

[ Parent | on HN ]
Re:Llama.cpp AI Performance with the GeForce RTX 5090 Review by perching_aix on Monday March 10, 2025 @07:30AM (#43319343)
Saw a video from Bolt Graphics, a startup trying to do that among other things. Supposedly they'll have demos at several conventions later this year, like at Hot Chips.

I think it's an incredibly tall order to get into the GPU game for a startup, but it should be good entertainment if nothing else.

[ Parent | on HN ]
Re:Llama.cpp AI Performance with the GeForce RTX 5090 Review by someothherguyy on Monday March 10, 2025 @07:47AM (#43319491)
https://www.youtube.com/watch?v=8m-gSSIheno

Sounds like hype

[ Parent | on HN ]
Re:Llama.cpp AI Performance with the GeForce RTX 5090 Review by someothherguyy on Monday March 10, 2025 @07:38AM (#43319409)
> yet companies don't want that

I think there are quite a few constraints you are glossing over here.

[ Parent | on HN ]
Re:Llama.cpp AI Performance with the GeForce RTX 5090 Review by KronisLV on Monday March 10, 2025 @08:11AM (#43319715)
Oh, I definitely am! It's always cool when someone with domain specific knowledge drops by and proceeds to shatter that dream with technical reasons that are cool nonetheless, the same way how LPCAMM2 doesn't work with every platform either and how VRAM has pretty stringent requirements.

That said, it's understandable that companies are sticking with whatever works for now, but occasionally you get an immensely cool project that attempts to do something differently, like Intel's Larrabee did, for example.

[ Parent | on HN ]
Re:Llama.cpp AI Performance with the GeForce RTX 5090 Review by Cane_P on Monday March 10, 2025 @08:37AM (#43319918)
SOCAMM will enter production in the end of the year:

https://www.tomshardware.com/pc-components/dram/nvidia-repor...

The CEO of SK Hynix have confirmed that it is in the works:

https://www.mk.co.kr/en/it/11245259

And big companies have started to analyze it, to figure out how it will fit into their domain:

https://x.com/Jukanlosreve/status/1892916771692421228?t=3ikB...

[ Parent | on HN ]