News for hackers. Stuff that matters.
Re:Effective Tensorflow by 0xbear on Sunday August 13, 2017 @03:36PM (#15004507)
But _why_ use TF when you have PyTorch which is just as powerful, runs noticeably faster for most workloads, _and_ is easy to understand? What are you gaining by using TF these days?
Re:Effective Tensorflow by jamesmishra on Sunday August 13, 2017 @04:11PM (#15004680)
I think at the moment, TensorFlow has a better tooling and deployment story with tools like TensorBoard[1], Tensor Processing Units in Google Cloud[2], distributed training[3], and deploying to mobile apps[4].

As time passes, PyTorch will probably get better at all of these things, but the core team and surrounding community need to make it a priority.

Meanwhile, Google's investment in TensorFlow will continue, and there's no sign that it will be a bad bet for large-scale deployments... even if PyTorch becomes the favored system for prototyping.

[1]: https://www.tensorflow.org/get_started/summaries_and_tensorb...

[2]: https://www.servethehome.com/google-cloud-tpu-details-reveal...

[3]: https://www.tensorflow.org/deploy/distributed

[4]: https://www.tensorflow.org/mobile/

Re:Effective Tensorflow by 0xbear on Sunday August 13, 2017 @05:02PM (#15004914)
TPU will be everywhere next year in the form of NVIDIA Tesla V100. And FB is clearly getting much better ROI. It's not even close.
Re:Effective Tensorflow by david-gpu on Sunday August 13, 2017 @07:02PM (#15005465)
Google's TPU and Nvidia's GPUs are competing products.

Source: works at NVidia.

Re:Effective Tensorflow by 0xbear on Monday August 14, 2017 @12:15AM (#15006547)
Why the downvotes? V100 does have a TPU-like unit in it.
Re:Effective Tensorflow by Q6T46nT668w6i3m on Sunday August 13, 2017 @05:50PM (#15005146)
While I’ve observed PyTorch running faster for my (convolutional-based) research, it’s within a few milliseconds of TensorFlow and Keras. If this type of difference mattered (it might for some uses), I would imagine you’d use CuDNN directly. I guess that’s the point, the libraries are all wrapping the same library. It’s like measuring the IO performance between programming language standard libraries (they should all be close to the speed of the underlying system call).
Re:Effective Tensorflow by 0xbear on Monday August 14, 2017 @05:00AM (#15007374)
We just plain can't do data augmentation quickly enough with TF. Queues-schmeyes, doesn't matter. Still tops out at about 35MB/sec on MS COCO and starves even a single Titan Xp. On the same hardware, with the same data augmentation steps, PyTorch gets ~50MB/s or so and saturates the GPU, since it never has to wait for data. In fact it can even read faster than that, and automatically parallelize the forward pass across several GPUs. You do still retain full control over placement, however. Super slick.
Re:Effective Tensorflow by mamp on Sunday August 13, 2017 @04:21PM (#15004730)
It's nice to have the Keras front-end for TF - less boilerplate etc.

I don't think there would be a reason why Keras couldn't sit on PyTorch but I can't see any work on this.

Keras is very productive unless you are doing NN research. Also you can also run it using CNTK if you want speed up RNNs which are slower on TF compared with CNTK.

Re:Effective Tensorflow by Q6T46nT668w6i3m on Sunday August 13, 2017 @05:37PM (#15005103)
I use Keras for machine learning research and find that it works extremely well. The one advantage of PyTorch is more flexible broadcasting, but that introduces other problems.

A major advantage of Keras is that if you’re using a standard training scheme (e.g. training a convolutional neural network for image classification), your research will be entirely focused on either the underlying architecture (that’s easily summarized and serialized), your custom layers (my favorite abstraction for convolutional neural networks), and losses. Keras’ abstractions eliminate stuff like IO that I find extremely distracting.

Re:Effective Tensorflow by niyazpk on Sunday August 13, 2017 @08:31PM (#15005868)
Can you please expand on what you mean by custom layers and abstractions and stuff? Thanks!
Re:Effective Tensorflow by amrrs on Monday August 14, 2017 @05:16AM (#15007430)
Nice video explaining a bit on the same https://www.youtube.com/watch?v=bvZnphPgz74
Re:Effective Tensorflow by programnature on Sunday August 13, 2017 @05:47PM (#15005142)
If you want to run in production.

Even FB doesn't use PyTorch in production, and instead uses Caffe2.

Re:Effective Tensorflow by Q6T46nT668w6i3m on Sunday August 13, 2017 @05:52PM (#15005156)
PyTorch was certainly used in production at Facebook (and elsewhere).

It might’ve changed in the past few months as Caffe 2 matured, but there were a few internal applications that used PyTorch.

Re:Effective Tensorflow by programnature on Sunday August 13, 2017 @05:56PM (#15005170)
Fwiw this is what Soumith has said: "Internally at Facebook, we have a unified strategy. We say PyTorch is used for all of research and Caffe 2 is used for all of production."

https://www.oreilly.com/ideas/why-ai-and-machine-learning-re...

Its not exactly a secret that PyTorch's tradeoffs favor research and not production.

Re:Effective Tensorflow by Q6T46nT668w6i3m on Sunday August 13, 2017 @06:08PM (#15005219)
Totally. And I think it’s a solid strategy. However, there’s certainly an interest (internally and externally) in providing a better inter-operability story between the two. I imagine something like using Keras for model creation (and possibly training) and running (either on mobile or the cloud) on some Caffe 2 deployment.
Re:Effective Tensorflow by programnature on Sunday August 13, 2017 @06:20PM (#15005270)
Its a good strategy. But its no silver bullet either. If you're exporting to a "static graph" platform, your losing a major benefit of PyTorch. If you mostly just care about shipping to production, a case can be made to just use tf/caffe2/mxnet etc from the start.

While PyTorch is extremely cool, the fanboyism is out of hand, thinking that what's good for their corner of the universe must be awesome for every use case and therefore TF is a overcomplex turd. Its not like the people designing these systems are stupid.

Re:Effective Tensorflow by Q6T46nT668w6i3m on Sunday August 13, 2017 @06:26PM (#15005304)
I agree. PyTorch’s dynamism is fantastic. However, I have no idea how you’d manage to recompile PyTorch code to Caffe 2 in a satisfying way. If something is released, I suspect it’d be limited to a subset of PyTorch features (I’d also bet that subset doesn’t include the features that make PyTorch compelling).
Re:Effective Tensorflow by tinyrick2 on Monday August 14, 2017 @03:08AM (#15007038)
Some people (like me) use Windows. PyTorch is not supported by Windows currently.
Re:Effective Tensorflow by H1Supreme on Monday August 14, 2017 @09:01AM (#15008280)
Spin up a virtual machine? It's worth getting into Linux for machine learning.
Re:Effective Tensorflow by CodesInChaos on Monday August 14, 2017 @09:31AM (#15008424)
Passing GPUs to VMs is tricky, especially with nvidia consumer GPUs. It's certainly not "just spin up a VM".
Re:Effective Tensorflow by jorgemf on Sunday August 13, 2017 @07:29PM (#15005583)
I think the are 2 main reasons: TF is backed by Google and you can use for research and production (server & mobile) with minor changes. You can also train your TF models in the cloud without having to setup anything (as in Google Cloud[1] or TensorPort[2])

[1] https://cloud.google.com/ml-engine/

[2] https://tensorport.com/

Re:Effective Tensorflow by make3 on Sunday August 13, 2017 @04:47PM (#15004852)
I really don't think the speed thing is as clearcut as you let it out to be
Re:Effective Tensorflow by jjawssd on Sunday August 13, 2017 @03:39PM (#15004516)
Tensorflow is more documented and talked about right now which is why more people will be interested in it
Re:Effective Tensorflow by 0xbear on Sunday August 13, 2017 @03:51PM (#15004584)
I'm not actually sure it's "more documented". It has more documentation, that's true, but that's not the same thing as better documentation, and after some point the mountain of documentation becomes a daunting obstacle. TF IMO is far beyond that point.
Re:Effective Tensorflow by p1esk on Sunday August 13, 2017 @06:09PM (#15005224)
TF documentation is excellent. PyTorch, on the other hand, felt like almost entirely undocumented. A month ago I tried to implement something fairly trivial (Karpathy's character RNN model), and there was no documentation at all on how to run it on multiple GPUs. I had to go through unrelated models code to understand how this could be done (using external library DataParallel). This might have improved since.

Also compare the number of TF and PT questions on SO.

Re:Effective Tensorflow by SimbaOnSteroids on Sunday August 13, 2017 @04:28PM (#15004771)
I find the documentation daunting but I had assumed that was because I didn't really understand how AI works and that as my understanding grew the docs would start to make more sense.
Re:Effective Tensorflow by 0xbear on Sunday August 13, 2017 @05:00PM (#15004901)
Take a look at PyTorch. Much easier to understand, fewer bugs, much less obscure all around. I've spent years working with TF and have come to dislike its inflexible, verbose, and convoluted way of doing things. I'm 80% on PyTorch now.
Re:Effective Tensorflow by mamp on Sunday August 13, 2017 @06:27PM (#15005310)
> I've spent years working with TF

Ha, TF was only released in Nov 2015 but sitting around waiting for networks to finish training makes it feel longer than that!

Re:Effective Tensorflow by 0xbear on Sunday August 13, 2017 @08:49PM (#15005960)
I worked with it _before_ it was released, and with DistBelief before that. For a long time I considered TF to be the best option for the class of problems I work on. I was also extremely skeptical of PyTorch, but it won me over. Now that I'm no longer on a steady diet of Google Kool-Aid, I have the luxury of picking the best tool for the job. For a number of things that tool is made by Google (protocol buffers, gtest, gmock, grpc, some other things). But not for ML.
Re:Effective Tensorflow by mamp on Monday August 14, 2017 @06:48AM (#15007748)
Based on your experience I'll definitely give PyTorch a go. Thanks.
Re:Effective Tensorflow by Q6T46nT668w6i3m on Sunday August 13, 2017 @05:42PM (#15005123)
I like and use both PyTorch and TensorFlow. You’re entirely correct that TensorFlow is far more verbose than PyTorch, but Keras remedies this (for many problems). I’d also add that PyTorch has a narrower scope than PyTorch and because of this introduces problems when scaling or distributing (PyTorch’s flexible broadcasting isn’t without trade-offs). Maybe there’ll be a convergence of Caffe 2 and PyTorch to remedy that problem.
Re:Effective Tensorflow by dr1337 on Sunday August 13, 2017 @06:25PM (#15005292)
For me, Keras is the least verbose of Tensorflow and PyTorch, however I find it too sparse and there's too much "magic" happening under the nicely done API. I have found PyTorch to be the happy half-way point between Tensorflow and Keras in terms of verbosity.
Re:Effective Tensorflow by Q6T46nT668w6i3m on Sunday August 13, 2017 @06:31PM (#15005327)
I’m curious, what parts do you find too magical? I’ve been trying to help with this problem (clarifying or surfacing features that are too magical without relinquishing Keras’ commitment to “convention over configuration”), so your (or others) feedback is welcome!
Re:Effective Tensorflow by make3 on Sunday August 13, 2017 @04:49PM (#15004865)
Tensorflow was the first modern tool supported by a large ai company, so probably mostly historical reasons. Also, Google is likely still the company with the largest AI departments, and the company that gives them the most visibility.
Re:Effective Tensorflow by kuschku on Sunday August 13, 2017 @04:08PM (#15004666)
Is PyTorch actually faster than TensorFlow with the C++ static compilation?
Re:Effective Tensorflow by 0xbear on Sunday August 13, 2017 @04:56PM (#15004893)
I've tried XLA and observed no measurable benefit for most models, small benefit for some, and worse performance for others. That was a few months ago, so maybe things have changed since then. It won't be a huge difference no matter what they do, because the vast majority of time is spent in CUDNN and gemm/gemv anyway
Re:Effective Tensorflow by kuschku on Sunday August 13, 2017 @05:35PM (#15005094)
What I'm interested in is AMD support, actually, w.r.t. GPU performance.
Re:Effective Tensorflow by dr1337 on Sunday August 13, 2017 @06:22PM (#15005280)
There's currently an open PR on GitHub for AMD GPU support https://github.com/pytorch/pytorch/pull/2365