Large language models are having their Stable Diffusion moment

Re:Large language models are having their Stable Diffusion m... by BulgarianIdiot on Monday March 13, 2023 @05:12PM (#35142745)

GPUs have too many cores, and each of them needs to read and write to RAM really fast, so any such indirection introduced through interfaces like DIMM is an obstacle to performance.

I suspect the future is tightly integrated CPU+GPU+RAM, but we may STILL have DIMM slots to acts as a buffer between "fast" RAM and disk. So your swap file goes from "fast RAM" to "DIMM RAM" before it has to go to disk, basically.

[ Parent | on HN ]

Re:Large language models are having their Stable Diffusion m... by dheera on Monday March 13, 2023 @05:28PM (#35143034)

Conceptually that's already the case i.e. L1 cache -> L2 cache -> RAM -> swapfile

If the definition of RAM is DIMMs then it's really just a matter of merging the CPU and GPU into one unit with a massive L2 cache (or maybe we have an L3 cache) of e.g. 32GB that's able to hold a massive model without cache misses.

[ Parent | on HN ]

Re:Large language models are having their Stable Diffusion m... by jandrese on Monday March 13, 2023 @07:18PM (#35144634)

Seems like AI type work is more memory constrained than per-core performance constrained.

[ Parent | on HN ]

Re:Large language models are having their Stable Diffusion m... by bitRAKE on Monday March 13, 2023 @11:52PM (#35147234)

Long term is fuzzy, but the 3D chip tech seems poised to offer exactly that: massive cache, GPU and CPU cores. NVidia isn't going to play nice because they don't like where this positions them in this future. AMD, Intel and Apple don't seem to need NVidia. This tension is bad for end-users.

[ Parent | on HN ]