But Karpathy is completely right; students who understand and internalize how backprop works, having implemented it rather than treating it as a magic spell cast by TF/PyTorch, will also be able to intuitively understand these problems of vanishing gradients and so on.

Sure, instead of "the problem with backpropagation is that it's a leaky abstraction" he could have written "the problem with not learning how back propagation works and just learning how to call a framework is that backpropagation is a leaky abstraction". But that would be a terrible sub-heading for an introductory-level article for an undergraduate audience, and also unnecessary because he already said that in the introduction.

I never disagreed with the utility and importance of understanding backprop. I'm glad the article exists. And it could be easily improved -- and all of us can gain [1] by acknowledging this rather than circling the wagons [2], so to speak, or excusing unforced errors.

> ... he could have written "the problem with not learning how back propagation works and just learning how to call a framework is that backpropagation is a leaky abstraction". But that would be a terrible sub-heading ...

My concern isn't about the heading he chooses. My concern is deeper; he commits a category error [3]. These following things are true, but Karpathy's article gets them wrong: (1) Leaky abstractions only occur with interfaces; (2) Backpropagation is algorithm; (3) Algorithms can never be leaky abstractions.

Karpathy could have communicated his point clearly and correctly by saying e.g.: "treating backprop learning as a magical optimization oracle is risky". There is zero need for introducing the concept of leaky abstractions at all.

---

Ok, with the above out of the way, we can get to some interesting technical questions that are indeed about leaky abstractions which can inform the community about pros/cons of the design space: To what degree is the interface provided by [Library] a leaky abstraction? (where [Library] might be PyTorch or TensorFlow) Getting into these details is interesting. (See [4] for example.) There is room for more writing on this.

[1]: We can all gain because accepting criticism is hard. Once we see that even Karpathy messes up, we probably shouldn't be defensive when we mess up.

[2]: No one is being robbed here. Criticism is a gift; offering constructive criticism is a sign of respect. It also respects the community by saying i.e. "I want to make it easier for people to get the useful, clear ideas into their heads rather than muddled ones."

[3]: https://en.wikipedia.org/wiki/Category_mistake

[4]: https://elanapearl.github.io/blog/2025/the-bug-that-taught-m...

Hear hear, one of my favorite comments recently.

Can’t agree more about the technical points (category error etc), and then the unexpected switch to the value of receiving constructive criticism as a gift not an attack.

Myself, I’m definitely conditioned to receive it as an attack. I’m trying to break this habit. This morning I gave some extensive feedback to some friends who have a startup. The whole time I was writing it, I was stressing out that they’d feel attacked, because that’s how I might take similar criticism.

How was it actually received? A mix I think. Some people explicitly received it as a gift, and others I’m not so sure.

> often I find his writing and speaking to be more than imprecise

I think that's more because he's trying to write to an audience who isn't hardcore deep into ML already, so he simplifies a lot, sometimes to the detriment of accuracy.

At this point I see him more as a "ML educator" than "ML practitioner" or "ML researcher", and as far as I know, he's moving in that direction on purpose, and I have no qualms with it overall, he seems good at educating.

But I think shifting the mindset of what the purpose of his writings are maybe help understand why sometimes it feels imprecise.

Whoever chose this topic title perhaps did him a disservice in suggesting he said the problem was backprop itself, since in his blog post he immediately clarifies what he meant by it. It's a nice pithy way of stating the issue though.

Nah, Karpathy's title is "Yes you should understand backprop", and his first highlight is "The problem with Backpropagation is that it is a leaky abstraction." This is his choice as a communicator, not the poster to HN.

And his _examples_ are about gradients, but nowhere does he distinguish between backpropagation, a (part of) an algorithm for automatic differentiation and the gradients themselves. None of the issues are due to BP returning incorrect gradients (it totally could, for example, lose too much precision, but it doesn't).

Yeah - he chose it as a pithy/catchy description of the issue, then immediately clarified what he meant by it.

> In other words, it is easy to fall into the trap of abstracting away the learning process — believing that you can simply stack arbitrary layers together and backprop will “magically make them work” on your data.

Then follows this with multiple clear examples of exactly what he is talking about.

The target audience was people building and training neural networks (such as his CS231n students), so I think it's safe to assume they knew what backprop and gradients are, especially since he made them code gradients by hand, which is what they were complaining about!