(0,0,0), A, and B are the three points of a right-triangle.
Hmm... okay, yeah. I see what you mean. Every time I think about how to do things, its linear algebra time.
The thing is: I've had this feeling before in other maths: the more you study one math subject, the "worse" you get in another. There's surprising things you can do with algebra alone without any calculus... but once you learn calculus, you kinda forget those old algebra tricks.
Your next line suggests you already know this, but just so it's clear it's not, they are three points in an arbitrary triangle. Each vector has a right triangle between the origin and (two of?) it's axis, but I'm not sure that's helpful.
Yeah, I agree in that it feels like you should be able to do this, but I just know that I've been stumped in the past, and can't figure it out now either.
sin(a) / A = sin(b) / B = sin(c) / C
Apparently, MY trigonometry is awful and needs more practice! Law of sines is basic 'non-right triangles' stuff.
There are three legs: A-0, B-0, and B-A. Hmmm... B-A seems like a vector operation (but its a really easy one, so... probably can be figured out without formally studying linear algebra).. That gives us the "A", "B" and "C" for lengths.
Aaaannnnd I'm stuck again. I have the lengths but I don't have the angles. I only need to find one angle and the other two are solved with law of sines.
Maybe a system of equations at this point (lol, another subject where linear algebra helps). The law of sines gives 3-unknowns, but only 2-equations. So I still need one more constraint before I can math-out a solution.
------
EDIT: Law of sines is fully:
sin(a) / A = sin(b) / B = sin(c) / C = d
Where "d" is the diameter of the circumcircle of the triangle. So the circumcircle's diameter gives us the 3rd equation needed to solve the above system of equations (using non-matrix math).
So yeah, I think its possible. Its just waaayyyy harder to do. I also am handwaving a difficult part. I guess... take the average of 3 points, which would give the center of the circumcircle. Then sqrt(x, y, z) from the center to find the radius, then 2*radius = d.
Wow, that's a lot of work compared to an easy projection matrix...
You could also use the law of cosines to get a triangle's angles pretty easily when you know its edge lengths: https://mathworld.wolfram.com/LawofCosines.html.
I mean, I recognize that I'm just "reinventing cross-product" here to get the length of opposite-leg and the direction of adjacent-leg. But it does seem feasible to do all of these things in Trigonometry alone.
A little bit of Linear algebra (cross product creates an orthogonal vector) would definitely make this problem easier.