I have great respect for the PyTorch-TPU team, but I would recommend not heavily advertising PyTorch-TPU support until this major feature disparity is made up.
https://pytorch-lightning.readthedocs.io/en/latest/tpu.html#...
I just read the linked page and found no references to data loading limitations or performance limitations. Is it only in the video which isn't search indexed and few people would bother watching?
edit: The page literally advertises the speed of TPUs with "In general, a single TPU is about as fast as 5 V100 GPUs!" which is the exact opposite of warning people.
Maybe that's the way you feel, but for me that's very different. Pytorch is much more than just an API (which is also nothing to scoff at).
It's also much cleaner documentation, a very different ecosystem of libraries (mostly better, but sometimes lacking depending on the niche), less magic (which makes it easier to debug). It also has the benefit of less ecosystem churn, while the transitions of TF1->2 as well as the external Keras->internal Keras are a shitshow that's almost as bad as Python2->3.
So you can't blame the PyTorch team. If there's anyone to blame, it's Google Cloud. In the meantime, I don't think there's any harm advertising PyTorch with TPU support if running on TPUs with PyTorch is often much faster than running on GPUs with PyTorch.
TPU support is VERY real... but yes, sometimes it breaks but PyTorch and Google are working very hard to bridge that gap.
But we have dedicated partners at Google on the TPU team working to get Lightning working seamlessly on pods.
Check out the discussions here: https://github.com/PyTorchLightning/pytorch-lightning/issues...
TPU support is real. Pytorch does in fact run on TPUs. But you don’t support TPU CPU memory, the staging area that you’re supposed to fill with training data. That staging area is why a TPU v3-512 pod can train an imagenet resnet classifier in 3 minutes at around 1M examples per second.
You will not get anywhere near that performance with pytorch on TPUs. In fact, you’re expected to create a separate VM for every 8 TPU cores. The VMs are in charge of feeding the cores. That’s insane; I’ve driven TPU pods from a single n1-standard-2 using tensorflow.
Repeat after me: if you are required to create more than one VM, you do not (yet!) support TPU pods. I wish I could triple underline this and put it in bold. People need to understand the limitations of this technique. Creating 256 VMs to feed a v3-2048 is not sustainable.
Picture a person with one arm and without legs. Would you say they aren’t “1:1 in terms of features”? They certainly won’t be winning any races.
And unlike real people, you can’t graft on a prosthetic limb to help this situation. The issue I’m describing here is a fundamental one that everyone keeps trying to sweep under the rug and pretend isn’t an issue. And then everyone wonders what’s going on.
We just need to be a part of the effort to help bridge the big gap and barriers keeping users from TPU adoption.
https://pytorch-lightning.readthedocs.io/en/latest/tpu.html#...
This was mentioned above, but nowhere on that page does it talk about any limitations whatsoever.
We’re not talking about a small 10% reduction in performance here. We’re talking like 40x differences.
If it seems unbelievable, and like it can’t possibly be true, well: now you understand my frustration here, and why I’m trying to break the myth.
Notice not a single benchmark has ever gone head to head in MLPerf using pytorch on TPUs. And that’s because using pytorch on TPUs requires you to feed each image manually to the TPU on demand, from your VM. Meaning the TPU is always infeed bound.
Engineers should be wincing at the sound of that. Especially anyone with graphics experience. Being infeed bound means you have lots of horsepower sitting around doing nothing. And that’s exactly the situation you’ll end up in with this technique.
There’s a way to settle this decisively: train a resnet classifier on imagenet, as quickly as possible. If you get anywhere near the MLPerf v0.6 benchmarks for tensorflow on TPUs, I will instantly pivot the other direction and sing the praises of pytorch on TPUs far and wide.
Luckily, that form is changing. There are interesting plans. But they are still just plans.
It’s better to go the other direction, I think. I ported pytorch to tensorflow: https://twitter.com/theshawwn/status/1311925180126511104?s=2...
Pytorch is mostly just an api. And that api is mostly python. When people say they “like pytorch”, they’re expressing a preference for how to organize ML code, not for the set of operations available to you when you use pytorch.