News for hackers. Stuff that matters.

A comment on Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?. Parent comment

Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by torginus on Friday August 08, 2025 @06:01PM (#44842128)
No it's not slow - a single NAND chip in SSDs offers >1GB of bandwidth - inside the chip there are 100+ wafers actually holding the data, but in SSDs only one of them is active when reading/writing.

You could probably make special NAND chips where all of them can be active at the same time, which means you could get 100GB+ bandwidth out of a single chip.

This would be useless for data storage scenarios, but very useful when you have huge amounts of static data you need to read quickly.

[ Parent | on HN ]
[dead]
Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by torginus on Saturday August 09, 2025 @04:31AM (#44844949)
That bandwidth is for the whole GPU, which has 6 chips. But anyways, what I'm proposing isn't for the high-end and training, but for making inference cheap.

And I was somehat conservative with the numbers, a modern budget SSD with a single NAND can do more than 5GB/s read speed.

[ Parent | on HN ]
Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by torginus on Saturday August 09, 2025 @04:31AM (#44844953)
That bandwidth is for the whole GPU, which has 6 mermoy chips. But anyways, what I'm proposing isn't for the high-end and training, but for making inference cheap.

And I was somehat conservative with the numbers, a modern budget SSD with a single NAND can do more than 5GB/s read speed.

[ Parent | on HN ]