News for hackers. Stuff that matters.
Re:Practical Deep Learning for Coders by throwaway2016a on Tuesday December 20, 2016 @05:53PM (#13225070)
I know it says graduate level math not required but it does need some math correct? What is the minimum?
Re:Practical Deep Learning for Coders by jph00 on Tuesday December 20, 2016 @05:58PM (#13225112)
You need to have a basic understanding of matrix products (from linear algebra) and the chain rule (from calculus). Both can be learnt on demand using Khan Academy - no need to go study math first, but just study what you need when you need it. We give brief explanations of both when they come up.

The wiki (http://wiki.fast.ai) has links to necessary learning resources for each lesson.

Re:Practical Deep Learning for Coders by bogomipz on Tuesday December 20, 2016 @06:19PM (#13225294)
So these are the two subsets of those maths needed for this course but in general is Deep Learning largely based in just Calculus and Linear Algebra?

Are there any others you would recommend learning and/or brushing up on in a more general sense for ML/DL?

Thanks for making this available by the way!

Re:Practical Deep Learning for Coders by jph00 on Tuesday December 20, 2016 @06:35PM (#13225395)
Deep learning is pretty much entirely linear algebra and calculus, at a mathematical level. But I really think that a code-first approach is much better. So I'd suggest just getting started working through the course, and really dive into the Jupyter notebooks for each lesson. Do lots of experiments.
Re:Practical Deep Learning for Coders by bogomipz on Tuesday December 20, 2016 @07:57PM (#13225895)
>"But I really think that a code-first approach is much better."

This is really refreshing to hear Most of the books I have looked at seem to be mostly math-first and very densely so at that. As much as I appreciate math it's not always the warmest welcome.

I am looking forward to working through your course and seeing part 2 as well. Cheers.

Re:Practical Deep Learning for Coders by Cybiote on Tuesday December 20, 2016 @08:41PM (#13226138)
If you want a solid understanding of DL⊂ML, you'll also need a good understanding of probability theory and information theory. While the mechanics require just matrix math and some vector calc, you'll also want to know linear algebra proper and these days, function spaces especially. A small helping of analysis, for measure theory, will not hurt either.
Re:Practical Deep Learning for Coders by bogomipz on Tuesday December 20, 2016 @09:11PM (#13226270)
Interesting that you mentioned Information Theory, when I think of this I think of Shannon, encoding symbols, bauds, bit, noise and channels in the Telecom network sense.

But I guess this all applies quite similarly in the neural network sense in DL/ML as well?

Re:Practical Deep Learning for Coders by jph00 on Tuesday December 20, 2016 @09:15PM (#13226291)
There's a 1:1 relationship between likelihood functions and entropy, so a background in information theory is likely to make your understanding of loss functions come quicker.
Re:Practical Deep Learning for Coders by bogomipz on Wednesday December 21, 2016 @12:34AM (#13227025)
Thanks for the insight.
Re:Practical Deep Learning for Coders by jph00 on Tuesday December 20, 2016 @09:13PM (#13226281)
I strongly disagree. This has been necessary for many previous approaches, but it's not fundamental to the topic.

Sometimes reading and understanding papers requires that background, I guess.

Re:Practical Deep Learning for Coders by Cybiote on Wednesday December 21, 2016 @02:14AM (#13227380)
First of all, let me say that what you all have put together is truly excellent. It covers the sort of things toy treatments and papers leave out or gloss over if you actually want to get something working in the real world.

That said, I strongly disagree with your disagreement. There was a recent paper whose abstract I read that made me think of homotopies between convolutional networks. Unfortunately I lost the paper behind some stream and never got to read it proper. In the context of that, I realized that the search for convnet design will likely soon be highly automatable, obsoleting much of the work that many DLers are doing now.

What will be future proof is understanding information theory so that loss functions become less magical. Information theory is needed to understand what aspects of the current approaches to Reinforcement learning are likely dead ends (typically to do with getting a good exploration strategy, also related to creativity). Concentration of measure is vital to understanding so many properties we find in optimization, dimensionality reduction and learning. Understanding learning stability and ideal properties of a learner/convergence means being comfortable with concepts like Jacobians and semi-positive definiteness for a start.

Probability theory is needed for the newer variational methods, whether in the context of autoencoders or a library like Edward (whose like I think is the future). Functionals and the variational calculus is becoming more important, in both deep learning and for understanding the brain. There's lots of work in game theory of dynamical systems (think evolutionary game theory) that can help contextualize GANs as a special case of a broader category of strategies.

Much to the contrary, the topics I mentioned are both the future of deep learning and future proof in general. This blog post by Ferenc captures my sentiment on the matter: http://www.inference.vc/deep-learning-is-easy/

Re:Practical Deep Learning for Coders by jph00 on Wednesday December 21, 2016 @04:27AM (#13227840)
I agree with all you say here. And indeed for researchers interested in building the next stage of deep learning, such topics may be important.

However this course is about teaching how to use deep learning today to solve applied problems. Most participants in the in-person course worked in business, medicine, or life sciences. It is to that audience that my comment was directed.

So we're perhaps each guilty of assuming the person asking the question wishes to study deep learning in the way we are focused on ourselves. :) Hopefully between the two of us we have now answered the question more fully than either of us individually!...

Re:Practical Deep Learning for Coders by spenuke on Wednesday December 21, 2016 @09:53AM (#13229546)
This exchange is pretty interesting. Cybiote – do you think it's possible for a person to do interesting work without formal graduate-level education?
Re:Practical Deep Learning for Coders by Cybiote on Wednesday December 21, 2016 @01:17PM (#13231303)
Yes, I certainly do think it possible. But it will be more difficult and slower going. Read widely, as the biggest hurdle for a self-learner will be in identifying and filling their gaps of knowledge. Read papers so you learn how to use language such that you are not setting off any crank or dangerous dabbler alarms. Reading lots is also necessary to habituate yourself to the jargon. Read the books by David MacKay and Christopher Bishop. Don't waste time if you're stuck on papers, it means either you're lacking either the shared knowledge or the authors are being purposely obtuse. Save anything you think is the prior for later (unless is a minor iteration of something, which most are). It might take multiple readings spread out over a large time span before you can truly understand the important concepts and papers.

If you can, try and organize or join study groups where the levels of skill are varying. Having a group will help in those times when it seems all too much and you're ready to quit.

Finally, don't try to compete with the well heeled industry titans and their GPU Factory Farms. Find an understudied but important niche where your lack of knowledge is not so much a setback because even if available mental tools will differ, everyone is equally ignorant on the terrain.

Re:Practical Deep Learning for Coders by deepnotderp on Tuesday December 20, 2016 @07:04PM (#13225568)
How the hell is deep learning graduate level math?
Re:Practical Deep Learning for Coders by jph00 on Tuesday December 20, 2016 @07:17PM (#13225645)
Re:Practical Deep Learning for Coders by deepnotderp on Wednesday December 21, 2016 @01:22AM (#13227177)
There's no way that's graduate level math. I've gone through Nando's lectures and although they're more mathematically involved than most courses, they aren't grad level math.
Re:Practical Deep Learning for Coders by jph00 on Wednesday December 21, 2016 @04:29AM (#13227848)
IIRC the course details suggest it is most appropriate for a graduate level. I may be misremembering however.
Re:Practical Deep Learning for Coders by master_yoda_1 on Wednesday December 21, 2016 @05:32AM (#13228110)
You might be right. The level of education is different for every individual and also people come from different country.

In America you can be a graduate in computer science without knowing calculus, and in some other country they teach calculus/probability in high school.

So the above person might have studied the chain rule and entropy in high school. So its not exactly graduate level math for everyone.

Re:Practical Deep Learning for Coders by vinay427 on Wednesday December 21, 2016 @08:53AM (#13229057)
> In America you can be a graduate in computer science without knowing calculus, and in some other country they teach calculus/probability in high school.

This certainly isn't true at my university in the US. As a matter of fact, I'm not sure why you named a country at all seeing as this varies by university. Anyway, I would definitely agree that on the scale of reputable international universities this is not grad-level math. A graduate school in a well-qualified university would expect students to either know this material or be able to learn it on their own. They are intro classes for the math major at my (and many other) universities.

Re:Practical Deep Learning for Coders by learc83 on Wednesday December 21, 2016 @04:12PM (#13232666)
>In America you can be a graduate in computer science without knowing calculus

I don't know how. Every CS program I've ever seen requires calculus.

>and in some other country they teach calculus/probability in high school

Practically every high school in America offers calculus. Not everyone is required to take it, but most of the students going on to study CS do.

Re:Practical Deep Learning for Coders by deepnotderp on Sunday December 25, 2016 @07:34PM (#13256246)
Even the average students here take calculus in HS....
Re:Practical Deep Learning for Coders by deepnotderp on Monday December 26, 2016 @03:33PM (#13259642)
Just to confirm, by entropy you mean shannon entropy, right? If so, yeah.
Re:Practical Deep Learning for Coders by thanatropism on Wednesday December 21, 2016 @08:39AM (#13228954)
My academic masters' program still had a couple of classes on data science and mathematical finance. So besides their research people are able to work outside academia.