I think it's maybe something like 13.8 'credits' an hour on Colab, and you get 500 credits for $50 straight up, or $50 a month (I'm truly a sucker for simple flat pricing schemes with a natural cap on them, it's good for the overzealous network trainer's/developer's wallet! :D). So that's like, I dunno, $1.38 per hour for an A100 basically guaranteed (not bad at all! And the H100 is coming soon, I'd assume! :D)

If training takes ~9.91-9.96 seconds, and we ignore everything else in the process (assuming we have some kind of strange Elvish magical computers that don't require any spinup of any sorts)... then that's (9.91 to 9.96)/60/60 * 1.38 = 0.0037988 - 0.0038180 dollars per run, or .37988 - .38180 cents per run. The full setup including install from clone, data download, and network init, I'd estimate being lower bounded at maybe 1.2-1.3 cents per run or so with a good internet connection (but I'm not entirely sure about that! D:). Upper bound for a reasonably fast machine I think would be no more than 2 cents, clean start to finish for a single training run. Multiples for best-of (maybe not the safest idea), or better yet -- simple ensembling of the EMA-ed models could be upper bounded at likely no more than ~4 cents or so for 5 models, if I'm doing my math correctly.

That said, the 'cents' calculation likely I think is .37988 - .38180 cents in this case.

What's weird is that that does seem a bit steep considering it's 8 V100s for 45 seconds, and those were...pretty pricy at that time, I think? So maybe something is horribly wrong with my math! D:

Hope that helps, great question and many thanks for the question, happy to answer any follow-up questions you might have. This is a very interesting line of inquiry, and I haven't yet spent enough time developing it yet! :D