I get what you're saying, but complex numbers go right into literally any slot a real number goes into. That isn't true of a matrix. Plus, they implicitly include the reals as a subset, which lets them follow in the tradition of the integers, rationals, and reals. It's sort of hazy, but I'd only start to draw the line at the p-adiacs.

It is a bit hazy, yes. For me having a total order relationship is one of the intrinsic properties of intuitive numbers. Complexes don't have that, but reals do, so I'm sure if you meant you can put them in the slot, but you'd compare the modulus.

That's why I'd prefer to relabel them like "2-d numbers" for instance, to make it clear that some properties are affected like going from points on a line to points on the plane.

That's a super fair point. I guess to me it matters more that it has all the usual field behaviors with the reals as a subset, but it is the first step "up" the hierarchy where you lose something instead of just gaining. I'm sort of used to substituting a metric in for order, but that obviously doesn't work everywhere.

Should we also stop calling negative numbers numbers?

Maybe. Consider for example Hossack (2020) Knowledge and Philosophy of Number. He indeed lays out a framework in which negative "numbers" aren't considered numbers, precisely because negative numbers combine a magnitude with a direction.

Let's throw out every textbook and start over, then.

Maybe we should. Irrespective of any concerns on the philosophical underpinnings of mathematics, there is an active discourse on the shortcomings of pedagogical resources. See for example Hung-Hsi Wu and his negative view of "Textbook School Mathematics (TSM)". His critique is aimed at the US. I am unfamiliar with any cohesive argumentation across for example European curricula.