News for hackers. Stuff that matters.
Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by nodja on Saturday August 09, 2025 @06:42AM (#44845466)
Loading here refers to loading from VRAM to the GPUs core cache, loading from VRAM is extremely slow in terms of GPU time that GPU cores end up idle most of the time just waiting for more data to come in.
Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by frabcus on Wednesday August 13, 2025 @04:21AM (#44885869)
Thanks, got it! Think I need a deeper article on this - as comment below says you'd then need to load the request specific state in instead.