you can set your max spot price high, and/or you can periodically save the trained weights (to an EBS volume) and continue from where you left off, same as recovering from a crash. Probably not a huge deal.

Here's how you do it with Tensorflow:

https://www.tensorflow.org/versions/r0.10/how_tos/variables/...

The sweet spot would be updated AWS GPU instances at spot prices. You could spin up XX instances for a relatively short time to train fast. Needs some Kubernetes etc cluster setup for distributed training but should be doable. If the spot price is $0.1-$0.2/hr that's pretty attractive, considering that is close to the price of electricity alone