The Gibbs formulation of 3D vector analysis/calculus was certainly useful, but forgets most of the spatial structure. Using Clifford Algebra (Geometric Algebra) fixes everything.
Almost all of the value in GA comes from the use of multivectors and the wedge product. It's very hard to find a justification for the "geometric product", and there's no real geometric interpretation of it. Therefore treating it as fundamental is a difficult-to-justify choice. "Exterior Algebra" on its own has everything you want without the objectionable parts.
I suppose you could even go through the relevant laws of physics and only identify the units for quantities that need to be added together (or to make the quantities inside more complex functions unit-less).
In the case of torque and energy you basically run into the problem that torque times angle is an amount of work and angle is traditionally unit-less. You could however give angle a unit by introducing some 'unnecessary' conversion factor like 'degrees per radian', which is only used for trigonometry. In that case you could reformulate torque as 'energy per angle'.
Possibly of interest, as mentioned in Appendix C of https://www.nist.gov/publications/architecture-software-assi... there is a start to this in https://en.wikipedia.org/wiki/ISO/IEC_80000