The average of the 3 points would give you the triangle's centroid, not necessarily its circumcenter. You can still calculate the circumradius from a, b, and c: https://mathworld.wolfram.com/Circumradius.html.
You could also use the law of cosines to get a triangle's angles pretty easily when you know its edge lengths: https://mathworld.wolfram.com/LawofCosines.html.
sin(a) / A = sin(b) / B = sin(c) / C
Apparently, MY trigonometry is awful and needs more practice! Law of sines is basic 'non-right triangles' stuff.
There are three legs: A-0, B-0, and B-A. Hmmm... B-A seems like a vector operation (but its a really easy one, so... probably can be figured out without formally studying linear algebra).. That gives us the "A", "B" and "C" for lengths.
Aaaannnnd I'm stuck again. I have the lengths but I don't have the angles. I only need to find one angle and the other two are solved with law of sines.
Maybe a system of equations at this point (lol, another subject where linear algebra helps). The law of sines gives 3-unknowns, but only 2-equations. So I still need one more constraint before I can math-out a solution.
------
EDIT: Law of sines is fully:
sin(a) / A = sin(b) / B = sin(c) / C = d
Where "d" is the diameter of the circumcircle of the triangle. So the circumcircle's diameter gives us the 3rd equation needed to solve the above system of equations (using non-matrix math).
So yeah, I think its possible. Its just waaayyyy harder to do. I also am handwaving a difficult part. I guess... take the average of 3 points, which would give the center of the circumcircle. Then sqrt(x, y, z) from the center to find the radius, then 2*radius = d.
Wow, that's a lot of work compared to an easy projection matrix...