News for hackers. Stuff that matters.
Re:Practical Deep Learning for Coders 2022 by jph00 on Thursday July 21, 2022 @09:18PM (#32187342)
Hi folks - I'm the creator/teacher of this course. I'd be happy to answer any questions that you have about the course, learning deep learning in general, or the state of deep learning in 2022.
Re:Practical Deep Learning for Coders 2022 by knicholes on Thursday July 21, 2022 @10:36PM (#32187826)
Thank you, so much, for enabling us mortals the power of ethical, modern AI. Your work, and the work of your colleagues, has brought so much good to this world.

It wasn't until the last few years I saw people start freezing the model and just fine tuning the last layers. I've watched presenters from flamingo and imagen talk about their similar approaches. I heard it here first, at fastai.

Re:Practical Deep Learning for Coders 2022 by jph00 on Thursday July 21, 2022 @10:41PM (#32187857)
Haha yes - it's been wonderful to see how (eventually) the deep learning world has taken to transfer learning!
Re:Practical Deep Learning for Coders 2022 by iTokio on Friday July 22, 2022 @01:36AM (#32188883)
What’s your take on meta learning?
Re:Practical Deep Learning for Coders 2022 by nindalf on Friday July 22, 2022 @04:16AM (#32189774)
I tried the 2021 course but I didn't finish. I think the biggest friction for me was using the remote machine. I wasn't able to make steady progress like I do with my offline learning projects.

How far away is the fast.ai from working on a Mac? PyTorch recently gained support (https://pytorch.org/blog/introducing-accelerated-pytorch-tra...) but that's only the start. Is this something that is being worked on?

Re:Practical Deep Learning for Coders 2022 by jph00 on Friday July 22, 2022 @04:49AM (#32189948)
The good news is that every lesson in this course is actually run on Kaggle Notebooks, which is a free cloud environments including GPUs. So you don't need to set up anything and it runs on any computer with a modern web browser!

Mac support for all the libs used in the course will probably continue to improve in the coming months and there should be no reason you won't be able to run the stuff for the course locally on a Mac at that time. Having said that, even the M2 trains deep learning models much slower than even the free NVIDIA GPUs provided by Kaggle. So you'd only want to use local development for the smallest and simplest models. (The course shows how to train models that are fairly cutting edge and some take a while to train even on modern GPUs, so they wouldn't be a good fit for a Mac.)

Re:Practical Deep Learning for Coders 2022 by uwuemu on Sunday July 31, 2022 @01:57PM (#32297252)
I think last time I tried this, I kinda gave up as soon as it got to the point of hand-waving hardware and telling you to run notebooks on a third party's web service. How is that democratizing AI? It's the very opposite! Not good. What is the intended audience here? Uni-level students will learn most of these within their programs if they're interested in AI. So they're not really it. Is the intended audience "coders"? If so, most of these "coders" will have to somehow get their employer on board (typically a corporate entity very much NOT interested in "coding" something in a 3rd party ecosystem) or do it themeselves. Hence, I want to take an RTX 3080+, 64+ gb of ram, big ass SSD and I want to get through the training. Not learn some basics on somebody else's platform (come on, even from the pov of OS, running both training AND notebooks on some 3rd party's private platform is so against the idea of open source...) and call it a day. What use is that? That may be enough if you want to be a cog in somebody else's machine, but not if you want to do something useful by yourself (I say "may" because big tech generally isn't interested in your "mad AI skillz" unless you also have a student loan backed piece of paper proving you successfully learned that for the last couple of years).

There will always be smart individuals and talented small teams that can successfully integrate AI into their products, but it's not thanks to the courses like this.

If you're going to aim at coders, there has to be clear path demostrated from the beginning to the end. From starting up your first notebook on your local dev machine and running training on your local training machine to setting up inference in the final app (.net app or whatever)

Re:Practical Deep Learning for Coders 2022 by nindalf on Friday July 22, 2022 @08:24AM (#32191180)
Thanks Jeremy, I'll give it another go.
Re:Practical Deep Learning for Coders 2022 by sooheon on Friday July 22, 2022 @02:34AM (#32189188)
I remember you were bullish about Swift a few years ago. What's your current view on non-python deep learning?
Re:Practical Deep Learning for Coders 2022 by jph00 on Friday July 22, 2022 @02:56PM (#32195793)
I'm disappointed that Google shut down the Swift for Tensorflow project, because I do think Swift is a great option for deep learning.

In some ways Jax is almost "non-python deep learning" since it's treating Python more like a DSL for the XLA backend. Normal Python code doesn't work in Jax. It's a pretty reasonable compromise since you still get all the benefits of the Python ecosystem.

Julia seems like it has the best foundations for deep learning, since everything can be written directly in Julia. But it doesn't have a great ecosystem as a general programming tool.

F# might turn out to be a good option.

Re:Practical Deep Learning for Coders 2022 by ngcc_hk on Friday July 22, 2022 @03:24AM (#32189466)
Interested that as well, especially the old school lisp to this new Ai.
Re:Practical Deep Learning for Coders 2022 by thefreeman on Friday July 22, 2022 @09:21AM (#32191618)
As someone who much prefers reading over watching videos, do you think I would miss much by just going through the book in the github repo? Or are those notebooks mainly supplemental to the videos.
Re:Practical Deep Learning for Coders 2022 by jph00 on Friday July 22, 2022 @02:52PM (#32195749)
I'd say it's the other way around - the videos are kinda supplemental to the book. The book has a lot more content, but doesn't have the interactive explanations in the course. Also the book is a couple of years old so is missing the more recent developments (but the principles haven't changed).
Re:Practical Deep Learning for Coders 2022 by sriram_malhar on Friday July 22, 2022 @02:42AM (#32189224)
Thank you so much for this course. I plan to go through it properly.

I have a search problem of my own and I have had a hard time applying what I have learnt (including the coursera DL specialization). The chief characteristics are: (a) It is a fuzzy search of a corpus that is in a non-English language. (b) The search should be able to run on a mobile phone _offline_.

Is this possible? Can training be done elsewhere and transferred to TinyML or some such? What would be a good forum to go seeking answers?

Re:Practical Deep Learning for Coders 2022 by leobg on Friday July 22, 2022 @06:15AM (#32190404)
Have you tried...

a) BM25 after some preprocessing (lemmatization etc.)

b) fastText / GloVe (possibly weighted by BM25)

The results can be surprisingly good. Often no need to bother with big language models or GPUs.

Re:Practical Deep Learning for Coders 2022 by sriram_malhar on Saturday July 23, 2022 @12:52PM (#32205074)
As far as I understand, BM25 is not for fuzzy searches. For a bit more context, the search terms are in the English script, but the words are basically the closest-sounding transcriptions of sounds in various Indian languages. Different people may render the same word differently in an English transcription. But there's enough crowd-sourced data to account for the ways in which words can vary.

For the same reason, GloVe is of not much use to me.

Re:Practical Deep Learning for Coders 2022 by leobg on Saturday July 23, 2022 @01:10PM (#32205288)
OK, I understand. That is a different problem indeed. Although I am not sure that BERT would be of much more help there either, unless you have quite a large training corpus. The simplest/cheapest approach might be some kind of transcription normalization. As you say, what is introducing ambiguity here is the act of transcribing these sounds into the English alphabet. There’s not a single source language/alphabet? Also, I am not sure if fastText/GloVe really would not pick up the semantic similarity in spite of differing transcriptions as long as you have a large enough training corpus. I’d experiment with the settings here (bag of words/skipgram, min/max lengths etc.).
Re:Practical Deep Learning for Coders 2022 by devnonymous on Friday July 22, 2022 @03:53AM (#32189626)
If the volume of data fits on a mobile phone for it to be offline, perhaps you don't need deep learning?
Re:Practical Deep Learning for Coders 2022 by sriram_malhar on Saturday July 23, 2022 @01:04PM (#32205207)
Perhaps. What alternative would you suggest? The search terms are fuzzy, and there are too many variants (not exactly misspellings) for me to encode them explicitly. So I thought I'd rather learn from a crowd-sourced corpus.

In my case, I don't want the model to be general. I can afford for it to be like a database index, tailored to that data.

Re:Practical Deep Learning for Coders 2022 by mkl on Friday July 22, 2022 @08:07AM (#32191062)
How many hours do you think this course would take for an experienced developer with plenty of applied maths but ~no machine learning?

How easy is it to do the course on my own hardware rather than cloud notebooks? Would that make it closer to practical deployment?

Re:Practical Deep Learning for Coders 2022 by jph00 on Friday July 22, 2022 @03:23PM (#32196078)
I'd suggesting budgeting about 80 hours for the course given that background. That should get you to a place where you can work on practical projects that are reasonably well within standard applications of deep learning.

Most practical deployment is done to cloud environments rather than local notebooks. The deployment exercise we do in the course is designed to show the key components you'll need for deploying simple models in practice.

Re:Practical Deep Learning for Coders 2022 by gandalfgreybeer on Friday July 22, 2022 @08:16AM (#32191131)
Not him nor will I talk about his course, but I’ve been in the field a reasonable amount of time (both on the academia and industry side). Honestly, applied maths will get you a long way and make it easier to digest the concepts (you might just see them as repackaged problems depending on your mileage). If you have good programming skills and discipline you practically have most of what you need.

Re the course, I just skimmed it and I think you can do most things on your own hardware but if you will actually use this for something practical (not just for you or a side project), being familiar with cloud tools is a big thing especially once you scale.

Re:Practical Deep Learning for Coders 2022 by qwrshr on Friday July 22, 2022 @11:42AM (#32193177)
out of curiosity, how much applied math should one bone up on? (Obviously the more the better, but diminishing marginal returns and all that.)
Re:Practical Deep Learning for Coders 2022 by gandalfgreybeer on Friday July 22, 2022 @08:41PM (#32198949)
Bare minimum is basic calculus, basic linear algebra and basic statistics. By basic, I probably mean first courses for those in most undergraduate programs.

I disagree with needing none and just going along as needed. That’s how you have machine learning models that look like they work but you don’t understand why they work so there might actually be problems.

Re:Practical Deep Learning for Coders 2022 by cinntaile on Monday July 25, 2022 @02:53PM (#32229214)
That's not quite what I said. I said to look things up if you don't understand, I made the assumption that the one asking has taken maths courses before. I interpreted bone up on as refreshing old knowledge but I could be wrong.
Re:Practical Deep Learning for Coders 2022 by cinntaile on Friday July 22, 2022 @01:57PM (#32195057)
None, just look things up as you go along if there is something you don't understand. You're likely not going to bother understanding how the optimization functions work or how the cost functions actually work anyway. They're implementation details in most cases.
Re:Practical Deep Learning for Coders 2022 by cweill on Friday July 22, 2022 @11:33AM (#32193082)
Hey Jeremy, i just want to say that I love your course and the way you teach. I refer everyone to the Fast AI in my YouTube videos on getting started with machine learning. Please keep up the great work!
Re:Practical Deep Learning for Coders 2022 by darepublic on Friday July 22, 2022 @12:53AM (#32188659)
Thanks for creating this fantastic content, I'm excited to give the 2022 course a look. It's an exciting time for AI. I'm curious about your thoughts on gpt3 and also the state of the art in computer vision, and object detection. All the best
Re:Practical Deep Learning for Coders 2022 by Oreb on Friday July 22, 2022 @08:37AM (#32191269)
Is there a new version of the book? All the links I find lead to the 2020 edition.
Re:Practical Deep Learning for Coders 2022 by jph00 on Friday July 22, 2022 @03:24PM (#32196094)
No, the book is continually updated for each reprint, but there isn't a separate edition.
Re:Practical Deep Learning for Coders 2022 by perfopt on Thursday July 21, 2022 @10:47PM (#32187895)
Thank you for creating this course. I started out on Tensor Flow but seeing this material I am in two minds whether I should abandon my TF book and start this one or save it for later. Most likely I am going to dive in :-)
Re:Practical Deep Learning for Coders 2022 by jph00 on Thursday July 21, 2022 @10:51PM (#32187911)
Both the Aurélien Géron and François Chollet TF books are absolutely terrific, and everything you learn from them will be extremely useful in becoming a deep learning practitioner, regardless of what framework you end up using. So if you've started with one of those books already, keep it up! :) The fast.ai course would actually be a pretty good addition to either book, since you'll get to see a whole different way of doing things, which might be useful to understanding what's going on.
Re:Practical Deep Learning for Coders 2022 by perfopt on Friday July 22, 2022 @01:07AM (#32188735)
Thank you
Re:Practical Deep Learning for Coders 2022 by rg111 on Friday July 22, 2022 @12:27AM (#32188524)
My suggestion would be to learn all the stuff from this course, using fast.ai library, and then gradually move towards PyTorch.

fast.ai is a fantastic educational resource and a great way to approach solving problems. But the library itself is lacking, and if you are an experienced programmer, when building real-life projects, you will be frustrated with fast.ai library.

The goal, IMO, should be learn from Jeremy Howard, s great instructor, communicator; learn his attitude, and then move to PyTorch (keeping the attitude, the knowledge, and the lessons with you.)

Re:Practical Deep Learning for Coders 2022 by mloncode on Friday July 22, 2022 @02:57AM (#32189308)
I am an experienced ML Engineer of 10 years and have worked at several large flagship tech companies. I do not agree that fastai is not appropriate for real-life projects. If you know the fastai library well, you know its a layered api on top of pytorch, which allows you to customize things to your needs quite easily. For example, it is fairly straightforward to get any pytorch model out of a Learner object. Furthermore, lots of care has been taken to keep the apis very consistent with pytorch as well.

It's also the only library I know of that consistently bakes in best practices like super convergence techniques or making things like test time augmentation very seamless. Many libraries lag behind fastai 1-2 years in this regards, and frankly it can be frustrating to use other frameworks sometimes.

There is a slight learning curve, for example to learn the DataBlocks API or the callback system, but once you really understand what is happening you will understand how nice the API is and how well engineered it is.

Side note: Regarding being an experienced software engineer, I highly recommend digging into how the python language was extended for this project (fastcore) and the development workflow used (nbdev), which I think could be interesting for those software engineers you mention as well as heighten your understanding of the ecosystem of tools.

Re:Practical Deep Learning for Coders 2022 by notpublic on Thursday July 21, 2022 @10:51PM (#32187914)
First of all, a big thanks!

What is your take on the current state of autonomous driving? Do you think we can achieve "full autonomy" with the technology we have currently?

Any new advances in DL that you are excited about?

Re:Practical Deep Learning for Coders 2022 by jph00 on Thursday July 21, 2022 @11:31PM (#32188179)
Honestly I'm not an expert on autonomous driving so I'm not sure I have great insights there. I do know quite a bit about computer vision however so feel qualified to comment on that bit -- I suspect the decision by Tesla to only use CV, and not LIDAR, may turn out to be a mistake. I don't see any reason why we couldn't achieve full autonomy with our current tech including LIDAR, although I don't know if it can be achieved at a practical latency and power budget.

The new advances in DL I'm excited about are things I show in the class: the accessibility of modern NLP thanks to the Hugging Face ecosystem; the power of ConvNeXt for even better computer vision models; the way Gradio and HF Spaces makes it trivially easy to get a working prototype application using DL online.

I'm also excited about hosted models and applications like GPT-3, DALL-E, and Codex. All the illustrations on our course website are from DALL-E, for instance!

Re:Practical Deep Learning for Coders 2022 by sabertoothed on Friday July 22, 2022 @08:08AM (#32191069)
I hope you stay as humble as you have been. But you're my personal hero. It is just incredible what you have done for the world.
Re:Practical Deep Learning for Coders 2022 by kachau on Saturday July 23, 2022 @04:22PM (#32207239)
Will it help me more or achieve more out of learning this course as compared to just directly using GPT-3 or Dall-E as paid user?
Re:Practical Deep Learning for Coders 2022 by fareesh on Friday July 22, 2022 @09:08AM (#32191501)
Hello Jeremy do you have any specific advice on tackling ASR using fast.ai?
Re:Practical Deep Learning for Coders 2022 by __rito__ on Friday July 22, 2022 @06:03PM (#32197644)
Jeremy, hi.

I have one question, and one only. Please answer:

Second part, when?

Re:Practical Deep Learning for Coders 2022 by baladre on Sunday July 31, 2022 @01:43PM (#32297170)

   En
Re:Practical Deep Learning for Coders 2022 by baladre on Sunday July 31, 2022 @01:43PM (#32297173)
.m
Re:Practical Deep Learning for Coders 2022 by baladre on Sunday July 31, 2022 @01:42PM (#32297168)

    .b