Re:Ask HN: How can ChatGPT serve 700M users when I can't run...
by nodja on Saturday August 09, 2025 @06:42AM
(#44845466)
Loading here refers to loading from VRAM to the GPUs core cache, loading from VRAM is extremely slow in terms of GPU time that GPU cores end up idle most of the time just waiting for more data to come in.
The limiting factor compared to local is dedicated VRAM - if you dedicate 80GB of VRAM locally 24 hours/day so response times are fast, you're wasting most of the time when you're not querying.