rkaplan

This is the correct answer. A key reason Rebco is so much better than older alternatives is that it can be cooled to superconductivity using only liquid nitrogen (77K), as opposed to liquid helium, which is much harder to work with.

Longtime DVC user here - this is going to be so helpful. We use DVC for all of our model and data versioning, but what's been missing is the ability to cleanly integrate that into our CI workflow. Looks like that's solved now! The cml.yaml syntax also looks quite nice, very easy to follow. Looking forward to trying this out.

Great anecdote from Jeff Dean's twitter [1] about how they thought to try this experiment, given that doctors didn't know such a thing was possible:

"Funny story. We had a new team member joining, and @lhpeng suggested an orientation project for them of "Why don't you just try predicting age and gender from the images?" to get them familiar with our software setup, thinking age might be accurate within a couple of decades,and gender would be no better than chance. They went away and worked on this and came back with results that were much more accurate than expected, leading to more investigation about what else could be predicted."

[1]: https://twitter.com/JeffDean/status/965720435290791936

This is not "AI that builds AI". The actual research behind AutoML is called NASNet (https://arxiv.org/pdf/1707.07012.pdf), and all it is simply: we found two good neural network layers (called NASNet normal cells / reduction cells in the paper) that work well on many different image datasets. It's a very cool research result. But it's not something that will replace AI researchers.

The AI Grant fellowship has been an awesome experience so far; I highly recommend applying if you're on the fence. The grant itself is useful (and this round includes $20k of Google cloud credits!) but equally valuable is the community that Nat has curated around the project. There are a lot of great folks working on AI from all sorts of places just a Slack message away. Now that Daniel Gross (YC's AI-focused partner) is involved as well, I'd expect that community to continue to grow.

This post doesn't even mention the easiest way to use deep learning without a lot of data: download a pretrained model and fine-tune the last few layers on your small dataset. In many domains (like image classification, the task in this blog post) fine-tuning works extremely well, because the pretrained model has learned generic features in the early layers that are useful for many datasets, not just the one trained on.

Even the best skin cancer classifier [1] was pretrained on ImageNet.

[1]: http://www.nature.com/articles/nature21056

"In contrast to deep neural networks which require great effort in hyper-parameter tuning, gcForest is much easier to train."

Hyperparameter tuning is not as much of an issue with deep neural networks anymore. Thanks to BatchNorm and more robust optimization algorithms, most of the time you can simply use Adam with a default learning rate of 0.001 and do pretty well. Dropout is not even necessary with many models that use BatchNorm nowadays, so generally tuning there is not an issue either. Many layers of 3x3 conv with stride 1 is still magical.

Basically: deep NNs can work pretty well with little to no tuning these days. The defaults just work.

This paper is important for a few reasons:

- Labeled data is very expensive. Historically attempts to learn on synthetic data has failed because ConvNets are very good at detecting small visual artifacts in the synthetic data and using those for classification during training. At test time on real data, those artifacts aren't present so model fails. A technique that can beat state-of-the-art (admittedly on a very narrow Eye Gaze dataset, but still) by only training on labels from synthetic data and testing on real data is important.

- They present a useful new idea to improve GAN training: using a history of "fake" images, rather than only the latest fake images from the generator. Ask anyone who has tried to train a GAN: the training is really unstable, each network only cares about beating the latest version of its "opponent". They show good improvements by saving many previous fake outputs to make the generator more robust. This reminds me of Experience Replay from DeepMind for RL.

- It's a published paper from Apple! Great that they are starting to contribute back to the research community.

This paper builds off of DeepMind's previous work on differentiable computation: Neural Turing Machines. That paper generated a lot of enthusiasm when it came out in 2014, but not many researchers use NTMs today.

The feeling among researchers I've spoken to is not that NTMs aren't useful. DeepMind is simply operating on another level. Other researchers don't understand the intuitions behind the architecture well enough to make progress with it. But it seems like DeepMind, and specifically Alex Graves (first author on NTMs and now this), can.

Highly recommend this video on the Deep Visualization Toolbox to anyone interesting in understanding more about how convnets work through visualization:

https://www.youtube.com/watch?v=AgkfIQ4IGaM

I'm in the class. He really said this. The CS183C videos are posted online about a week after each class, so you can watch it for yourself soon.