News for hackers. Stuff that matters.
Re:Who Invented Backpropagation? by pncnmnp on Monday August 18, 2025 @12:29PM (#44942471)
I have a question that's bothered me for quite a while now. In 2018, Michael Jordan (UC Berkeley) wrote a rather interesting essay - https://medium.com/@mijordan3/artificial-intelligence-the-re... (Artificial Intelligence — The Revolution Hasn’t Happened Yet)

In it, he stated the following:

> Indeed, the famous “backpropagation” algorithm that was rediscovered by David Rumelhart in the early 1980s, and which is now viewed as being at the core of the so-called “AI revolution,” first arose in the field of control theory in the 1950s and 1960s. One of its early applications was to optimize the thrusts of the Apollo spaceships as they headed towards the moon.

I was wondering whether anyone could point me to the paper or piece of work he was referring to. There are many citations in Schmidhuber’s piece, and in my previous attempts I've gotten lost in papers.

Re:Who Invented Backpropagation? by drsopp on Monday August 18, 2025 @12:52PM (#44942732)
Perhaps this:

Henry J. Kelley (1960). Gradient Theory of Optimal Flight Paths.

[1] https://claude.ai/public/artifacts/8e1dfe2b-69b0-4f2c-88f5-0...

Re:Who Invented Backpropagation? by pncnmnp on Monday August 18, 2025 @01:02PM (#44942867)
Thanks! This might be it. I looked up Henry J. Kelley on Wikipedia, and in the notes I found a citation to this paper from Stuart Dreyfus (Berkeley): "Artificial Neural Networks, Back Propagation and the Kelley-Bryson Gradient Procedure" (https://gwern.net/doc/ai/nn/1990-dreyfus.pdf).

I am still going through it, but the latter is quite interesting!

Re:Who Invented Backpropagation? by cco on Tuesday August 19, 2025 @04:06AM (#44949397)
Count another in the win column for the USA's heavy investment into basic sciences during the space race.

So sad to see the current state. Hopefully we can turn it around.

Re:Who Invented Backpropagation? by psYchotic on Monday August 18, 2025 @12:38PM (#44942573)
Re:Who Invented Backpropagation? by pncnmnp on Monday August 18, 2025 @12:42PM (#44942617)
Apologies - I should have been clear. I was not referring to Rumelhart et al., but to pieces of work that point to "optimizing the thrusts of the Apollo spaceships" using backprop.
Re:Who Invented Backpropagation? by costates-maybe on Monday August 18, 2025 @12:59PM (#44942826)
I don't know if there is a particular paper exactly, but Ben Recht has a discussion of the relationship between techniques in optimal control that became prominent in the 60's, and backpropagation:

https://archives.argmin.net/2016/05/18/mates-of-costate/

Re:Who Invented Backpropagation? by observationist on Monday August 18, 2025 @05:40PM (#44945640)
Kelley 1960 (the gradient/adjoint flight‑path paper) https://perceptrondemo.com

AIAA 65‑701 (1965) “optimum thrust programming” for lunar transfers via steepest descent (Apollo‑era) https://arc.aiaa.org/doi/abs/10.2514/6.1965-701

Meditch 1964 (optimal thrust programming for lunar landing) https://openmdao.github.io/dymos/examples/moon_landing/moon_...

Smith 1967 & Colunga 1970 (explicit Apollo‑type trajectory/re‑entry optimization using adjoint gradients) https://ntrs.nasa.gov/citations/19670015714

One thing AI has been great for, recently, has been search for obscure or indirect references like this, that might be one step removed from any specific thing you're searching for, or if you have a tip-of-the-tongue search where you might have forgotten a phrase, or know you're using the wrong wording.

It's cool that you can trace the work of these rocket scientists all the way to the state of the art AI.

Re:Who Invented Backpropagation? by duped on Monday August 18, 2025 @12:57PM (#44942797)
They're probably talking about Kalman Filters (1961) and LMS filters (1960).
Re:Who Invented Backpropagation? by pjbk on Monday August 18, 2025 @01:12PM (#44942982)
To be fair, any multivariable regulator or filter (estimator) that has a quadratic component (LQR/LQE) will naturally yield a solution similar to backpropagation when an iterative algorithm is used to optimize its cost or error function through a differentiable tangent space.
Re:Who Invented Backpropagation? by bgnn on Monday August 18, 2025 @03:11PM (#44944156)
So yeah, this was what I was thinking for a while. What about a more nonlinear estimator? Intuitively seems similar to me.
Re:Who Invented Backpropagation? by andyferris on Monday August 18, 2025 @05:59PM (#44945767)
I believe the reason it works in nonlinear cases is that the derivative is “naturally linear” (to calculate the derivative, you are considering ever smaller regions where the cost function is approximately linear - exactly “how nonlinear” the cost function is elsewhere doesn’t play a role).
Re:Who Invented Backpropagation? by bgnn on Tuesday August 19, 2025 @02:44AM (#44948891)
that makes a lot of sense actually. thank you.
Re:Who Invented Backpropagation? by cubefox on Monday August 18, 2025 @01:07PM (#44942924)
> ... first arose in the field of control theory in the 1950s and 1960s. One of its early applications was to optimize the thrusts of the Apollo spaceships as they headed towards the moon.

I think "its" refers to control theory, not backpropagation.

Re:Who Invented Backpropagation? by dataflow on Monday August 18, 2025 @12:41PM (#44942606)
I asked ChatGPT and it gave a plausible answer but I haven't fact checked. It says "what you’re thinking of is the “adjoint/steepest-descent” optimal-control method (the same reverse-mode idea behind backprop), developed in aerospace in the early 1960s and applied to Apollo-class vehicles." It gave the following references:

- Henry J. Kelley (1960), “Gradient Theory of Optimal Flight Paths,” ARS Journal.

- A.E. Bryson & W.F. Denham (1962), “A Steepest-Ascent Method for Solving Optimum Programming Problems,” Journal of Applied Mechanics.

- B.G. Junkin (1971), “Application of the Steepest-Ascent Method to an Apollo Three-Dimensional Reentry Optimization Problem,” NASA/MSFC report.

Re:Who Invented Backpropagation? by throawayonthe on Monday August 18, 2025 @01:01PM (#44942846)
it's rude to show people your llm output
Re:Who Invented Backpropagation? by dataflow on Monday August 18, 2025 @04:04PM (#44944703)
Wow, this is the first time I'm hearing such a thing. For clarity:

I pasted the output so a ton of people wouldn't repeat the same question to ChatGPT and burn a ton of CO2 to get the same answer.

I didn't paste the query since I didn't find it interesting.

And I didn't fact check because I didn't have the time. I was walking and had a few seconds to just do this on my phone.

Not sure how this was rude, I certainly didn't intend it to be...

Re:Who Invented Backpropagation? by epaulson on Monday August 18, 2025 @04:32PM (#44944976)
The 'it's rude to show your ai(llm) output' is a reference to this: https://distantprovince.by/posts/its-rude-to-show-ai-output-...
Re:Who Invented Backpropagation? by dataflow on Monday August 18, 2025 @05:02PM (#44945293)
What is "this" exactly? Is it a well-known author or website? Or otherwise a reference that one should be familiar with? It looks like a random blog to me... with an opinion declared as fact that's quite easy to refute.
Re:Who Invented Backpropagation? by actionfromafar on Monday August 18, 2025 @06:18PM (#44945891)
”This” is also the novel Blindsight.
Re:Who Invented Backpropagation? by cubefox on Tuesday August 19, 2025 @07:14AM (#44950312)
Re:Who Invented Backpropagation? by drsopp on Monday August 18, 2025 @01:02PM (#44942863)
Why?
Re:Who Invented Backpropagation? by danieldk on Monday August 18, 2025 @01:09PM (#44942956)
Because it is terribly low-effort. People are here for interesting and insightful discussions with other humans. If they were interested in unverified LLM output… they would ask an LLM?
Re:Who Invented Backpropagation? by drsopp on Monday August 18, 2025 @01:19PM (#44943051)
Who cares if it is low effort? I got lots of upvotes for my link to Claude about this, and pncnmnp seems happy. The downvoted comment from ChatGPT was maybe a bit spammy?
Re:Who Invented Backpropagation? by lcnPylGDnU4H9OF on Monday August 18, 2025 @01:54PM (#44943434)
> Who cares if it is low effort?

It's a weird thing to wonder after so many people expressed their dislike of the upthread low-effort comment with a down vote (and then another voiced a more explicit opinion). The point is that a reader may want to know that the text they're reading is something a human took the time to write themselves. That fact is what makes it valuable.

> pncnmnp seems happy

They just haven't commented. There is no reason to attribute this specific motive to that fact.

Re:Who Invented Backpropagation? by drsopp on Monday August 18, 2025 @02:42PM (#44943894)
> The point is that a reader may want to know that the text they're reading is something a human took the time to write themselves.

The reader may also simply want information that helps them.

> They just haven't commented.

Yes, they did.

Re:Who Invented Backpropagation? by Dylan16807 on Monday August 18, 2025 @06:29PM (#44945960)
> The reader may also simply want information that helps them.

The reader will generally want at least a cursory verification that it is information that helps, which dataflow didn't try to do.

Especially when you're looking for specific documents and you don't check if the documents are real. (dataflow's third one doesn't appear to be.)

Re:Who Invented Backpropagation? by drsopp on Tuesday August 19, 2025 @12:11AM (#44948189)
This I agree with completely.
Re:Who Invented Backpropagation? by bee_rider on Monday August 18, 2025 @03:33PM (#44944381)
Yours was a little bit more useful, it you essentially used the LLM as a search engine to find a real article, right?

Directly posting the random text generated by the LLM is more annoying. I mean, they didn’t even vouch or it or verify that it was right.

Re:Who Invented Backpropagation? by aeonik on Monday August 18, 2025 @01:29PM (#44943167)
I don't think it's rude, it saves me from having to come up with my own prompt and wade through the back and forth to get useful insight from the LLMs, also saves me from spending my tokens.

Also, I quite love it when people clearly demarcate which part of their content came from an LLM, and specifies which model.

The little citation carries a huge amount of useful information.

The folks who don't like AI should like it too, as they can easily filter the content.

Re:Who Invented Backpropagation? by mellosouls on Monday August 18, 2025 @05:10PM (#44945373)
See Widnall's overview here which discusses some of the ground that crosses over with what has come to be known as backpropagation:

The Minimum-Time Thrust-Vector Control Law in the Apollo Lunar-Module Autopilot (1970)

https://www.sciencedirect.com/science/article/pii/S147466701...

Re:Who Invented Backpropagation? by leokoz8 on Monday August 18, 2025 @07:37PM (#44946531)
It is in Applied Optimal Control by Bryson and Ho (1969). Yann LeCun acknowledges this in his 1989 paper on backpropagation:https://new.math.uiuc.edu/MathMLseminar/seminarPapers/LeCunB....

> "Since his first work on the subject, the author has found that A. Bryson and Y.-C. Ho [Bryson and Ho, 1969] described the backpropagation algorithm using Lagrange formalism. Although their description was, of course, within the framework of optimal control rather than machine learning, the resulting procedure is identical to backpropagation."

Re:Who Invented Backpropagation? by seertaak on Tuesday August 19, 2025 @02:23AM (#44948784)
Rumelhart et al wrote "Parallel Distributed Processing"; there's a chapter where he proves that the backprop algorithm maximizes "harmony", which is simply a different formulation of error minimization.

I remember reading this book enthusiastically back in the mid 90s. I don't recall struggling with the proof, it was fairly straightforward. (I was in senior high school year at the time.)