> - Why do we need complex numbers in the SE, what do they bring to the table and how does that tie in with physical intuition?

The complex numbers are the smallest algebraically closed field, they're the natural place for doing algebra.

> - How was the equation derived, intuitively?

I don't know if this how it was derived, but the way I see it it's pretty much the "square root" of the standard wave equation. So if you're trying to describe something where the solutions are mostly waves but you want to also include some other kinds of solutions, it's a very natural thing to try.

> - The laplacian in there feels like it's a "diffusion" term (as in the heat equation, things tend to spread and smoothen over time). Why isn't there ever a intuitive explanation in text books about this?

It's pure imaginary, so rather than a diffusion it's a rotation (phase change). So things tend to behave as waves with a frequency proportional to their energy.

> - [edit]: Many book trumpet that classical mechanics is just an approximation of QM, but unlike in special relativity where speed going to zero obviously gets you back to classical mechanics, I've never seen a clear example of a system where the quantum behavior "transitions" to classical behavior as a variable (e.g size) increases or decreases.

You tend to recover classical behaviour as energy and momentum go to infinity (so frequency goes to infinity and wavelength goes to zero). E.g. if you look at a double-slit interference pattern, as you increase the frequency/decrease the wavelength the classical bell curve becomes more and more visible.

> The laplacian in there feels like it's a "diffusion" term (as in the heat equation, things tend to spread and smoothen over time). Why isn't there ever a intuitive explanation in text books about this?

In an equation like

  df/dt = k Δf

the Laplacian create a diffusion-like effect only if k is a real positive number. If k is a real negative number, it creates the opposite effect that concentrates whatever f represent and make it more spiky.

When k is an imaginary number, it is not like diffusion o anti-diffusion (I don't know the technical name). The associated operator is unitary and the behavior is different.

Many books about the heat/diffusion equations and Schrodinger equation have this discussion. And the generalized versions where k is a functions that depends on x or t, or is a complex number instead of a real or complex number.

For an intuitive idea of what is happening, the idea is that you consider at time=0 the function

  f(x,t=0)=A sin(ωx).

If k is real and positive, the amplitude of the function decrease with time. And when ω is big, it decrease faster, so the small features disappear very faster than the smooth parts.

If k is real and negative, the amplitude of the function increase with time, and you have the opposite effect.

If k is imaginary, the function just rotate in the complex plane but does not increase or decrease the amplitude. So you get a wavy effect instead of a diffusion effect.

The initial equation is linear, i.e. there is no f^2 or something like that. With some hand waving you can extend the simple cases where f is sin or cos to more complex cases. The technical details are in https://en.wikipedia.org/wiki/Fourier_transform

Regarding your second to last point, have a look at https://en.wikipedia.org/wiki/Classical_limit