Ahh that looks like the error I was hitting, thanks. I might try replacing the bits as well, though I just upgraded pytorch from 0.1.12 to 0.3 and it became much slower (I killed it after 5-6 minutes of setup).
My fork is here: https://github.com/Yorwba/FastPhotoStyle
I was using the pytorch 0.1.12 installed with conda (following their USAGE.md) and it took ~30s total for the transfer.
I'm going to submit a PR, but it took me a bit of experimentation to fix these errors, so the code is a bit messier than I'd like.