> I've only ever known the Z axis to represent depth. Be it maths, physics or software development

A standard math textbook will usually show the x-axis pointing toward the reader, the y-axis pointing right, and the z-axis pointing up. Something like this:

    z
    |_
    /  y
   x

Similarly, if you're defining a cone or a spiral or what have you, you'll usually do it in terms of an r/θ/z coordinate system, where z is height. So the convention exists.

But obviously where the x and y axes are defined, the z axis will be perpendicular to the x-y plane.

Yes. Usually, the right-hand rule is followed. A positive rotation is from the positive x axis toward the positive y axis. If you curl the fingers of your right hand in that direction, then the thumb points along the positive z axis.