Quantization is a good idea, but I've had some bad experiences with bfloat16. (We couldn't get loss to decrease beyond a certain point when using bfloat16.) But that might have been an artifact of training. Still, I'd rather not harm the model if possible. It's hard to get a sense of perceptual quality, and I'm not convinced that validation tests are enough to capture the full nuances of GPT-2.
You're right that a single server would do the trick in this case. It's just so much easier to write a notebook with the logic in it than to set up a server, keep track of sessions, make read-eval-print logic, add error handling, etc. I was hoping to throw a couple hundred dollars at the problem of bandwidth hosting rather than spend time making a server.
A server also implies a fixed $300/mo cost, which is pretty expensive. After the initial demo, we'd probably end up turning the server off. I dislike the idea of the demo breaking after a month or two.
It's hard to quantify how much damage quantization does, but natural language generation is very subtle.
How many playing sessions a server with a single 2080Ti can support? Is it compute or memory bound? I'd plot num_sessions vs latency (time to compute a move), and estimate the costs for target scale/performance.