If every major language provides an O(n log n) sort function, is it still a leaky abstraction? I'd say no. You can use it without worrying much about the details.
But it sounds like the situation with back-propagation is different, since the internal details of the algorithm affect whether you get a usable answer at all.
A borderline case might be something like a SQL database, where in theory, creating an index shouldn't change query results, but in practice, the performance can change so much that it effectively does.
Unlike with back-propagation, you can tune database performance without worrying about queries returning different results (assuming they don't time out). So it's still a useful guarantee.
There are a lot of details to get right: how are elements compared? Is the sort stable? Is it efficient for small N? Is it efficient for nearly-sorted arrays? Is it efficient when almost all the elements compare equal? Is it guaranteed O(n log n) or average? If average, is there an input that reliably triggers n^2 behavior making it a DDoS vector?
Anything is a leaky abstraction when you care enough.
If the box is only labelled with O(nlog n) without specifying constants then there is a leak.
In much the same way, much of the interesting work with machine learning requires deep specialist knowledge in which selecting the best approach requires digging beneath the abstraction. Many specific applications have their own quirks, which is where significant gains are often made. (To be honest, I think the last year was more exciting for the advances in applied ML than in theoretical work.)
A co-worker wrote a library for distributed training of DNNs for a specialized use case. Because of certain quirks of our use case (think sparsity and consistent patterns in the data), there were certain optimizations he could make to the training process that gave nice linear scaling in the number of training nodes.
*incompetent - those who think they know something and are not willing to learn anything new
And then worst case? Best case? Average case? Which applies to random input? Nearly-sorted input? What's the overhead for a "short sort", can/should I use this to sort small sequences in a tight-ish loop or is it only for large sequences?
I was under the illusion that Big-O was always asymptotic complexity (ie worst case) and that other notations (little-o, big-omega, big-thetha etc) were used for best/average/etc case. Perhaps I'm wrong, however.
Which applies to random input? Nearly-sorted input? What's the overhead for a "short sort", can/should I use this to sort small sequences in a tight-ish loop or is it only for large sequences?
Indeed. The details matter.
My favourite example is how a naive linear search can perform better than a non-linear search with a better (in O() terms) algorithm if, for example, the linear search can do most of its work in cache (eg small input sizes or otherwise regular access patterns (predictable for prefetch)).
No, asymptotic notation only refers to the behaviour of the function (or algorithm) as you take the limit N->inf, e.g. changes to the size of the input. But the nature of the input often changes the behavior as well.
I was responding to a comment that I felt was needlessly dismissive. The default sort algorithm of any major language's standard library is usually not worth trying to beat, and calling someone "incompetent" for not reinventing the wheel each time they encounter data that needs to be sorted is absurd. Note: I said "usually", not "never". A competent programmer doesn't fix problems that don't exist, and you would absolutely have to profile code before convincing me that the choice of sorting algorithm was the source of a performance problem.
As far as the details go... In only a few languages (e.g. C) might you really even consider moving memory around rather than pointers. The semantics of standard library sort in Java, C#, Ruby, Python, Javascript... is sorting an array of primitives by value or objects by reference.
Great example :) You get different results if you use merge sort or quicksort.
The former is stable; the latter is not.
Okay...so you can be more specific with your API and specify whether stability is guaranteed. Eventually you can be specific enough that you essentially characterize the implementation.
Yes, if the performance is still not acceptable and you look into the problem and discover that your scenario could benefit from radix sort, or one of partial sorting, or some kind of intermittent sorting, all of which would require investigating the specific case at a "white box" level, ignoring the existence of a black-box O(n-log-n) sort API call.
I invite you to start sorting data which is often already sorted (or often all identical) and tell me you didn't need to worry about the details.
And neural networks using backpropagation to minimize an error function are definitely the former while the "go to" techniques of mathematics are generally the latter.
I think this is generally acknowledged, in fact.
Sooo it seems logical that students of these modern machine learning techniques should be impelled to "get their hands dirty" with the numericals processes involved since this will add to the data they use to drive the intuitions which allow them to avoid the multiple pitfalls of neural networks.
This is a non-trivial statement, because there are other things which are not leaky. For example, he's not arguing that deep learning practitioners should also learn assembly programming or go into how CUBLAS implements matrix multiplication. Although these things are nice to learn, you probably won't need them 99.9% of the times. Backprop knowledge, however, is much more crucial to design novel deep learning systems.
I would argue that it's not just for that. You need to understand what is happening inside DNN if you want to construct it properly.
The Authoritative Argument is not very convincing when your good authority over the course material being challenged. Maybe there is some irony or deeper truth, lost on me. Are you perhaps a leaky abstraction?
edit:
> The "code reuse as much as possible" mantra applies to when ...
... you are able to reasonably compromise, got it.
Neither is a straw man argument.
He wasn't making an argument from authority. He was essentially saying "because you are learning, not doing".
No, implicitly and very subtly. The essence of the argument was the student in a voluntary class. The implications are manifold, but not stating the implications has a meaning on the meta level, that the students understanding is not expected, but the students expectation of an authoritative answer has to be satisfied anyway by stating the obvious to establish or retain authority.
Maybe educational students should only require what's actually practical. That can include things that can be solved by computers if they teach ideas that have practical value.
Yes, but the logic goes the other way: if they are unable to use the technique manually, then one may conclude they do not understand the idea well enough to use it, whether or not Mathematica is available. The two, manual application and understanding/intuition, go hand in hand and can't be separated. My understanding is that this has even been studied more formally, there is a paper I remember reading by Kahneman and somebody else [1] about the development of intuition and the tension between heuristics-and-biases and naturalistic decision making. In short, don't trust people so much, if they say they understand something but are unable to actually do it, be a bit more skeptical.
[1] Found it: https://www.ncbi.nlm.nih.gov/pubmed/19739881 On second thought, it's probably not as relevant as I remembered it.
A better analogy might be stable sort versus unstable. If you need a stable sort, and unstable one is wrong.
(Not sure if that counts as an abstraction leak, though! It's just the semantics. Unstable sort doesn't leak that it is unstable; that's its behavior. If a sort is just documented as putting items in order, an actual implementation does leak information about whether or not it is stable.)
I would much rather respond to the students who ask about why they need to know the details, by saying "because you registered for this class."
Essentially the question "if you’re never going to write backward passes once the class is over, why practice writing them?" is another way of asking "why reinvent the wheel?". And the best answer is "because learning is all about reinventing the wheel". The "code reuse as much as possible" mantra applies to when you're "using" a technique to do something else, not when you're "learning" the technique itself. They might as well register for Calculus and ask "why learn integrals and derivatives when mathematica can do them for you", or take an Aerodynamics class and ask "why learn fluid mechanics and dynamics, heck newton's laws, when an airplane can run on autopilot." I doubt "because calculus is a leaky abstraction" or "because fluid dynamics is a leaky abstraction" is a good answer to that.