MikeBattaglia
Good point. Also, discovering exploits is also equivalent to the halting problem. So both things are impossible in the general case.
One way to look at this is to note that even with forward mode autodiff, there have generally historically been two different viewpoints for the whole thing, which I'll call the "computer science" view and the "algebraic" view.
The computer science view involves things that look computational graphs with nodes that look like {"value": 123, "deriv": 456}. We are storing real values along with propagated partial derivatives. We have a custom * operator which sets a * b = {"value": a.value * b.value, "deriv": a.value * b.deriv + b.value * a.deriv}. Other functions, like exp, sin, cos, log, etc are also extended to handle these kinds of input. There's a ton of literature that views things using this kind of framework going back to the 1960s.
The algebraic view uses dual numbers. Instead of writing {"value": 123, "deriv": 456}, we write 123 + 456h. We get the same results as the above: (a + bh) * (c + dh) = ac + (ad + bc)h. We can extend many functions in a natural way to the dual plane, such as exp, sin, cos, log, and get values there. There's also plenty of literature on these, going back to the late 1800s.
A modern view is to note that these two things are *identical*. It isn't that forward mode autodiff "uses" dual numbers; it is dual numbers. The set of elements of the form {"value": x, "deriv": y}, with addition and multiplication as stated, satisfies the axioms of a real algebra and is isomorphic to the dual numbers. We could have written {"real_part": x, "dual_part": y} if we wanted. You can see this viewpoint in some of the links I've posted elsewhere here.^[1]
So given all of that background, there are two answers to your original question. The first is to just simply view it as that "reverse-mode autodiff doesn't use dual numbers." Many people have this view, and I would say that that it really focuses on what I've called the "computer science" view above.
The second view is to note that the relationship between reverse-mode autodiff and dual numbers is the same as the relationship between reverse-mode autodiff and forward-mode autodiff. It would be silly to say that they are totally different, unrelated things: at the end of the day all we are really doing is changing the order in which we perform a bunch of multiplies of Jacobian matrices. I tend to view it as similar to the relationship between the DFT and the FFT: there is this super elegant linear algebra view involving DFT matrices. Do we say that the FFT "doesn't use matrices?" Well, I guess, but are we going to go so far as to say that it also doesn't involve linear algebra, etc? That is my view.
There are a few other differences between reverse-mode and forward-mode autodiff. Each individual operation in the computation graph, for instance, can be thought of as an individual instance of forward-mode autodiff. In reverse-mode, on the other hand, we typically store the (back-)propagated adjoints as additional data on the input node objects, not the output nodes. This is useful if we are thinking of backpropagation on a graph. It's up to you if you view these as involving materially different theories or just differences of implementation for the sake of optimization.
In short, the main thing is that there's less literature on a purely algebraic version of reverse-mode autodiff in general.
[1] There is one important difference: we often think of the dual numbers as an ordered ring with "h" infinitesimal. The first viewpoint doesn't use this part of the mathematical structure - and it's very interesting to note that it isn't even necessary! Although it's cute to think of h as infinitesimal, the autodiff properties we get flow purely from the algebraic properties of having h^2 = 0, regardless of order. Of course, though, we can always just think of the dual numbers as an unordered ring, if we want.
Here's the documentation: https://pytorch.org/tutorials/intermediate/forward_ad_usage....
> When an input, which we call “primal”, is associated with a “direction” tensor, which we call “tangent”, the resultant new tensor object is called a “dual tensor” for its connection to dual numbers[0].
Beyond that I'm not sure what to tell you, other than it's fairly easy to see that the dual numbers do satisfy the axioms of an ordered ring that I gave. Here's a large survey of various infinitesimal systems by Philip Ehrlich where he also notes the dual numbers are an ordered ring: https://arxiv.org/pdf/1808.03345.pdf.
The correct property is that if a ≤ b, a + c ≤ b + c, and if a ≥ 0 and b ≥ 0, then ab ≥ 0. It is fairly easy to see that these properties hold for dual numbers.
Pytorch does things slightly differently in that it is mostly focused on reverse-mode autodiff, and so it stores adjoints relative to the overall output rather than partial derivatives relative to the input, but this isn't really an entirely different thing, in the same way that the FFT isn't entirely different from the DFT.
There seems to be some confusion about the relationship between dual numbers and smooth infinitesimal analysis. Both have nilpotent elements, but with dual numbers the background logic is classical, whereas it isn't with smooth infinitesimal analysis.
https://towardsdatascience.com/forward-mode-automatic-differ...
Some thorough notes from MIT: https://book.sciml.ai/notes/08-Forward-Mode_Automatic_Differ...
Here is a teacher who taught a class using these, calling the dual element dx instead of h:
https://cornellmath.wordpress.com/2007/08/28/non-nonstandard...
Every element in the dual numbers is of the form a + bh, and in fact the entire ring can be turned into a totally ordered ring in a very natural way: simply declare h < r for any real r > 0. In essence, we are saying h is an infinitesimal - so small that its square is 0. So we have a non-Archimedean ring with infinitesimals - the smallest such ring extending the real numbers.
Why is this so important? Well, if you have some function f which can be extended to the dual number plane - which many can, similar to the complex plane - we have
f(x+h) = f(x) + f'(x)h
Which is little more than restating the usual definition of the derivative: f'(x) = (f(x+h) - f(x))/h
For instance, suppose we have f(x) = 2x² - 3x + 1, then
f(x+h) = 2(x+h)² - 3(x+h) + 1 = 2(x² + 2xh + h²) - 3(x+h) + 1 = (2x² - 3x + 1) + (4x - 3)h
Where the last step just involves rearranging terms and canceling out the h² = 0 term. Note that the expression for the derivative we get, (4x-3), is correct, and magically computed itself straight from the properties of the algebra.
In short, just like creating i² = -1 revolutionized algebra, setting h² = 0 revolutionizes calculus. Most autodiff packages (such as Pytorch) use something not much more advanced than this, although there are optimizations to speed it up (e.g. reverse mode diff).
The fractions that one gets from iterating the mediant process are exactly the "semiconvergents" one gets from the continued fraction process. Truncating the continued fraction gives the "convergents"; these are special mediants which are right before switching direction as one zig zags on the tree.
But at least they have the multiocular O, I guess.
Actually, the fact that people are supposed to pretend things are OK, while everyone over there is possibly a few steps away from being drafted into war, is another sign that things have headed in the wrong direction in exactly the manner OP is suggesting.
Is it really the position of mainstream archaeology that something like Gobekli Tepe was built by nomadic hunter-gatherer tribes who hadn't even invented the wheel? Some of the pillars weigh 22,000 lbs. There are intricate statues carved by people who supposedly didn't have metalworking or any metal tools. The entire thing is part of an even larger complex that has yet to be excavated and it was all planned out and coordinated by people who hadn't even invented writing or math. How plausible is this?
10 years ago, when these discoveries were much newer, mainstream archaeologists were still largely hung up on the idea that these ridiculous massive stone temples must somehow have been built by primitive hunter-gatherers. These days, it seems as though those attitudes have shifted, and that things like Gobekli Tepe call into question if we really have the right picture regarding the timeline of human development at all. It seems increasingly possible - even in mainstream archaeology - that humans 13000 years ago may simply have been further ahead than we thought.
One side effect of this paradigm shift is that people like Graham Hancock get to claim they were right all along. OK! Of course it this doesn't mean everything else he has ever claimed is right, but as far as I care, he can milk the "him vs the establishment" angle all he wants. He has a valid complaint - but more importantly, what institution is supposed to be able "debunk" him at a time when legitimate new discoveries are currently upending everything we thought we knew? You need to figure out what is actually going on first before you can hope to be Real Scientists "debunking" anything. Until you do, the notion that perhaps Gobekli Tepe was really built by people with a comparatively advanced level of civilizational development, rather than primitive hunter-gatherer cavemen, seems as decent a hypothesis to entertain as anything else. And if I were a betting man, I'd bet this really is the right hypothesis, in my view!
Here's what I think: modern Western civilization - to say nothing of historical colonial societies - has attempted to pulverize these cultures utterly: it has removed their spoken and written language, music and art; altered their general societal structure; created a false narrative that they are "savages"; and so on. Some of these events are very recent, and some people involved are even still alive today.
Then, we like to do this thing where we go ahead and "study" the same people whose culture we just destroyed, pretending all the while as if it were an ancient civilization destroyed thousands of years ago through some natural process.
Tens of millions of people are walking around in Africa today who use this language. Now that we have "discovered" this, maybe some day we will also "discover" that they have music, art, jokes, and an entire culture.
No shade on Ngom, though, as I am sure he is doing his best to bring some of these things to light within the Western academic system.
> Five hundred years ago, however, it would have made a very big difference. Before the advent of temperament systems, D-sharp and E-flat were two different notes. They weren’t just written differently; they sounded different. You can compare the historical versions of these notes yourself in this track I made... My track is tuned in a system called five-limit just intonation
^ These are not the historical versions of those notes. 5-limit just intonation was not in widespread use in Western Europe 500 years ago. 500 years is not before the advent of temperament systems. And so on. Teach this to your students however you think is best, but people on here may be interested to know that.
"My track is tuned in a system called five-limit just intonation via the magic of MTS-ESP. It’s the basis for all the tuning systems used in Western Europe between about 1500 and 1900."
No - at no point in the last 500 years was 5-limit just intonation ever the predominant tuning system used anywhere in Western Europe. The real predominant tuning system was called "meantone temperament," to which this article sadly devotes only about 3 words - and those words are only about 12 tone meantone keyboard layouts, not about the bigger, abstract idea of meantone temperament in general as it was understood and taught by practitioners of the day.
There is a very important difference between meantone and just intonation. The goal of meantone was to have four tempered perfect fifths (approximately a 3/2 frequency ratio) add together to approximate the fifth harmonic (or a 5/1 frequency ratio). Thus, the major third from the circle of fifths would approximate a 5/4 ratio with the tonic, and the major chord would approximate a very crunchy sounding 4:5:6 ratio. In order to do this, fifths are all flattened slightly to make the tradeoff - flattening the fifths by 1/4 of a "syntonic comma" was typical, or "quarter-comma meantone". Even though keyboard instruments evolved in a more well-tempered direction, meantone was the way that teachers of the common practice era (such as Leopold Mozart) still taught and thought about this stuff.
Meantone sounds noticeably different from just intonation, where the major third from the circle of fifths is a syntonic comma sharp of a 5/4 ratio (about 22 cents). In just intonation, if you want your major chords to be 4:5:6, you need to bring in this other, different, independent 5/4 major third that is not on the circle of fifths. As a result, certain chord progressions that are common in Western music will tend to exhibit strange sounding "comma drifts" if you play them in just intonation. Adam Neely has a good video on "Benedetti's Puzzle" about this for those who are interested.
Of course, there is nothing wrong with just intonation, and comma shifts can sound interesting if you want to deliberately use them in some kind of modern microtonal setting, but it simply isn't the tuning historically used in common practice Western music.
Anyway, though, if you go through the article with a marker and replace all instances of "just intonation" with "meantone," the general idea is mostly correct.