You can see this happening today. My favorite contemporary example is elliptic curves, which are a hot topic. Most introductory materials on elliptic curves will tell you that an elliptic curve is defined by an equation of the form: y^2 = x^3 + ax + b. Where did that come from? Why is there no x^2 term? How did anyone ever figure out that this particular equation would be good for cryptography? What other polynomial equations might be good for cryptography?
And then one day you learn about Montgomery curves, which do have an x^2 term but no x^0 term. So was the whole x^3 + ax + b thing a lie from the beginning? What about a complete cubic equation with all terms included? Is that an elliptic curve? Is it useful for cryptography?
Maybe my brain is just wired differently from most people, but those kinds of questions just pop up naturally for me, but I hardly ever see them either asked or answered. And this is true in every technical field I have ever studied, and I've had a very long and diverse career.
- There is not enough time. Unfortunately, formal courses have a fixed time, and the lecturer has to cover a certain amount of content in that time [1].
- Sometimes, its not actually possible to answer the 'why' questions with the amount of background that students have (or the lecturer has).
The above is also true for books/written down notes as well. For instance, in your example of elliptic curves, the answer to your questions about "why the form y^2 = x^3 + ax + b" requires students to understand algebraic geometry first, where such curves are defined in general (over arbitrary fields etc), rather than over the real field as you have. If you read such a book, you will see why in the general setting, its completely natural to define such curves. I don't know exactly why elliptic curves are good for cryptography, but I would hazard that the reason is perhaps only apparent when you study them in the abstract - hence why you don't see this explanation in your resources. The writer might not even know themselves.
[1] Yes yes, this is broken as well. In my ideal university there would be no semesters. Courses will be as short or as long as necessary to naturally end at an appropriate location.
I'm sad for you. All my professors did that in every lecture. I'd copy it down in a spiral notebook as he'd describe it step by step. Later, when reviewing the notes, it would bring back the lecture in my head.
This was before the days of videotape, but I still regret not taping the lectures with a cassette recorder.