It’s a lot simpler than you think, model serving is Horizontally scalable. You deploy the model running on a top of gpus each with a web server. Load balancer distributes the requests, probably a web socket streams the inference in chunks back to the client.