Well you'll be glad to know that PyTorch is available (in development form) on the TPU: https://github.com/pytorch/xla

If it's "available" but doesn't do anything useful, it's IMO not really "available". It's a good start though. I hope they don't throttle it in favor of TF because that'd only prolong TF's agony.

I think this is pretty unfair.

I know a bunch of PyTorch using people who have switched back to TF 2.0, and that is ignoring things like the Swift TF implementation.

The deployment story around TF is a lot better too.

Have you ever used Keras?

Yes I used Keras before PyTorch came out. IMO, Google needs to start over rather than try extensive wart removal on something that consists almost entirely of warts. These days Keras has subpar multi-GPU support (two GPUs are often slower than one), less flexibility, and more un-debuggable magic than PyTorch. And it doesn't look like things will be improving much (if at all) with TF 2.0.

And the TPU is a story of its own when it comes to usability. Take a look at their docs. TPU actually has to run on a separate machine, which you need to provision manually using their `ctpu` tool, and training of a trivial network on CIFAR looks like this: https://github.com/tensorflow/tpu/blob/master/models/experim...

Err - i don't get it. What alternative r u suggesting for development which is aimed at eventual production?

I am a complete noob, so i just did Keras (to TF) and GPUs.

If i switch to TPU - its still the same code.

When i get the budget to cluster, i think I am better off with one of the Deepmind libs which TF based.

No?

If you're just using off-the-GitHub models and pressing them into service, sure, Keras is adequate. If your work actually requires research and experimentation (as most practical problems on which actual cash money can be made do) you'll be much better served with PyTorch. The main cost in all this is not actually implementing or training your final model. That can be done in any framework. The cost is finding a model that works really well for a particular task. You will get there a lot sooner and with a lot fewer headaches if you use PyTorch for this more "researchy" part.

As to deployment, once you figure out what works, there's also libtorch, or if that doesn't work for you for whatever reason, implementing a model you know working hyperparameters for can be done in a day or two on whatever framework works for your backend, including TF.