News for hackers. Stuff that matters.
Re:Google/Trax – Understand and explore advanced deep learning by logicchains on Monday February 17, 2020 @03:00AM (#22346225)
>I would argue that this is not the fault of Tensorflow, but rather the hazard of being the first implementation in an extremely complex space. Seems like usually there needs to be some sacrificial lamb in software domains.

I'd agree if Google didn't have a history of building things with (arguably) unnecessarily complex APIs, like Angular1. I remember when Angular and React were new, seeing an Angular "cheatsheat" that was around 14 pages; the equivalent React cheatsheet was only 2 pages. Now, I do love the idea of Tensorflow 1, essentially a functional DSL to explicitly construct computation graphs, but Google's implementation of that idea was suboptimal: hard-to-follow error messages, not intuitive, multiple APIs to do the same thing (and continual API breakage as new APIs are introduced), difficult to debug. And even if the graph "compiled" correctly, it could still fail at execution time.

It's like they were building a programming language but lacked anyone with language design or PL theory background. Which makes sense given that anyone passionate about language design might prefer to work for somewhere closer to the cutting edge like Microsoft (C#, F#, Typescript, F*...), Facebook (Bucklescript, Hack), Apple (Swift) or Mozilla (Rust). Google does have its own languages, Dart and Go, but they're notable for ignoring and rejecting respectively cutting-edge PL theory (e.g. not disallowing null pointers, and in Go's case not even supporting parametric polymorphism). The day-to-day languages used at Google are also not particularly appealing to a PL enthusiast: Python, Java and non-modern C++.

Google software often also seems to care more about enforcing their idea of "best practices" on the user than about user experience. Tensorflow's C++ support is an example of this: it requires using Babel. As Babel doesn't easily support integration into an existing C++ project, this essentially means you have to change your whole project over to Babel just to use Tensorflow C++, which is a huge amount of effort to go to just to use a library. Especially when it's probably quicker to just rewrite the model in PyTorch, which provides a simple header file and static library for linking, the standard way of distributing C++ libraries. PyTorch also provides a nicer C++ API, because it's not confined to the ancient C++ standards that Google enforces, so can provide a modern API that's not full of macros (macros in modern C++ are considered bad practice; should only be used when there's absolutely no alternative).

To be fair, Google is really good at engineering language runtimes. Dart, Go and Tensorflow are all impressive works of engineering. They just seem to lack the organisational DNA for making them really nice to use, which maybe makes sense given the main source of their revenues is search/AdWords, the success of which is primarily driven by superior science/engineering. Compared to e.g. Facebook and Microsoft, that were/are in the business of making pretty things that users like to use (operating system, word processor, website). Or even comparing to Apple: people pay a huge premium for Apple phones over Android phones, in spite of their worse hardware, because of their more appealing design.

Re:Google/Trax – Understand and explore advanced deep learning by WnZ39p0Dgydaz1 on Monday February 17, 2020 @03:08AM (#22346253)
I would actually argue that the very first version of Tensorflow was great. I don't quite remember what version number that was, but it was well before 1.0. It was basically just a better version of Theano with a single way of doing things and very explicit graphs. I remember I ported all my Theano code without issues and loved it.

From then on it all went downhill. Lots of duplicate APIs were added and the code grew increasingly complex and opque because everyone at Google wanted in. Everyone wanted a slice of the pie to pad their resumes and internal performance reviews to be able to say "Look, I contributed this to TF! Promote me!" - Now it's a typical example of Conway's Law - a big mess that mirrors how messed up the internal incentive structure at Google is.

Re:Google/Trax – Understand and explore advanced deep learning by logicchains on Monday February 17, 2020 @03:22AM (#22346299)
Makes sense. I only started using it after it'd already been though tf.nn, tf.layers, and tf.estimators (which eventually got replaced by tf.keras, around the time I moved to PyTorch, and the time dropout was broken for two releases.. https://github.com/tensorflow/tensorflow/issues/25175). I remember though once spending half a day just trying to figure out why a graph that built correctly kept failing at execution time: I ended up having to binary-search the code changes because I couldn't find the cause of the error in the stack trace, something I've only experienced previously in template heavy C++ code. I don't imagine that situation was much better in 1.0.
Re:Google/Trax – Understand and explore advanced deep learning by WnZ39p0Dgydaz1 on Monday February 17, 2020 @03:29AM (#22346326)
The error message have always been an issue, even with early releases. That being said, it was much easier to trace errors when there were fewer high-level functions and you had to build the layers yourself. You fully understood what the graph looked like, and it was mostly obvious where something is likely to go wrong.

Sure, it's a bit more work, but in the long run it saves times because it avoids the situation where you need spend half a day trying to understand opaque TF code touched and extended by 100 different developers.

EDIT: Also, I LOVE the Github issue you posted. It's such a beautiful example of complexity cost when a large number of people is working on a project and there's no longer a single person that fully understands how code interacts across modules.

Re:Google/Trax – Understand and explore advanced deep learning by logicchains on Monday February 17, 2020 @03:50AM (#22346403)
I'd love to know what the solution is to prevent that eventually happening to a project, if one even exists. PyTorch seems to be doing better, but maybe that's just because it's a younger project. I'd be interested to see a blog post comparing Facebook's incentive structure for maintaining products to Google's.
Re:Google/Trax – Understand and explore advanced deep learning by thedance on Monday February 17, 2020 @01:03PM (#22349334)
It's a mistake to imagine that Google has some kind of centralized "programming languages" organization[1]. The people who designed Angular have probably never even met the people who designed Go, much less been influenced by them or shared influences. Same for TensorFlow. There may be "PL enthusiasts" at Google but the things they do at work are the same things everybody does at Google: moving data from one protobuf to another.

1: There is a "language platforms and tools" group (I used to work there) but its mission is stewardship of existing languages, not development of new ones.

Re:Google/Trax – Understand and explore advanced deep learning by 32gbsd on Monday February 17, 2020 @07:34AM (#22347237)
Exactly what I call the programmer open-source API trap. The moment I saw how tensor was built out and published almost instantly I knew it was a funnel for capturing innovation and would eventually be replaced when they found a better mouse trap. I did not spend much time learning that open source wheel. Hopefully this new one will be smaller.