Re:EVGA terminates Nvidia partnership [video]
by mnahkies on Saturday September 17, 2022 @09:05AM
(#32877213)
This is essentially what I have in mind - some kind of queue, with a mediator that starts/stops a spot instance using a custom image that has everything installed and configured to start on boot
With regards to state I just meant the image - wouldn't want to have to install everything at each start. Using something like aws or gcp I don't think this would be an issue, but I was window shopping some providers that specialize in GPU VMs and it wasn't clear if they provided equivalent services. Probably easiest to just use aws.
Re:EVGA terminates Nvidia partnership [video]
by stavros on Saturday September 17, 2022 @09:12AM
(#32877269)
I think if you couldn't make a ready-made image you'd use a Docker container with everything in it instead. Though then I don't know how you'd use spot instances, because AFAIK you want those for the "whenever it's available" startup and can't really be logging into each one to start them up. If they let you run a container, presumably they let you run other commands to set stuff up as well?
This kind of depends on the provider, but it's mostly an implementation detail. The main crux is a broker and a queue, I use Dramatiq (the site is written in Python) and it works great.
Re:EVGA terminates Nvidia partnership [video]
by mnahkies on Sunday September 18, 2022 @06:24AM
(#32886216)
With spot instances you just have to be prepared for them to get terminated at random. If you're processing messages off a queue in most cases you get this for free - the ack deadline will expire and the message will be redelivered to another subscriber.
Docker container might be a good shout, though I've not done anything using a GPU from docker before
Since this is a hobby project, I probably will never be able to break even on that. I'm thinking of writing a bit of code that will use GPU spot instances to perform the computation, and turn it off when the queue is emptyish.
You don't really need any state on the machines, the worker communicates with a broker, receives a very small bit of JSON containing the parameters, does the work and uploads the image to R2. Then, it sends a message back saying that image is done.
That way, you don't even care if you get interrupted mid-generation, since the work is idempotent. I can just spin up a machine whenever available and tear it down whenever.
It will be nice to try, though I think I'll use my desktop for a while yet due to a lack of users. That one takes around a minute to generate an image, and can't run while I'm using it.