If you follow through with the entire fastai course/book, then you will learn how to build an ML framework (the fastai library) from scratch. Yes, it starts out by saying "you can do deep learning with 4 lines of code" but it goes through all the inner workings, and talks about some pretty advanced stuff. Of course, no book can cover everything, but it also functions quite well as a jumping off point for more advanced self-study.