Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on LLaMA-7B in Pure C++ with full Apple Silicon support. Parent comment
An RTX 4090 has a memory bandwidth of 1,008 GB/s.
PCIe 4 x16 has a 32GB/s bandwidth.
DDR4 RAM has 3200 MT/s transfer rate.
An AMD 5900x can easily max that out.
A good NVME can hit 7 GB/s read speeds or better.
And the 4-bit CUDA kernels can pack 16x 4-bit ints into a single 64-bit transfer / register.
There are no comments on this comment yet.
An RTX 4090 has a memory bandwidth of 1,008 GB/s.
PCIe 4 x16 has a 32GB/s bandwidth.
DDR4 RAM has 3200 MT/s transfer rate.
An AMD 5900x can easily max that out.
A good NVME can hit 7 GB/s read speeds or better.
And the 4-bit CUDA kernels can pack 16x 4-bit ints into a single 64-bit transfer / register.