I never really used GLM, but Eigen was substantially slower than DirectXMath https://github.com/microsoft/DirectXMath for these things. Despite the name, 99% of that library is OS agnostic, only a few small pieces (like perspective projection matrix formula) are specific to Direct3D. When enabled with corresponding macros, inline functions from that library normally compile into pretty efficient manually vectorized SSE, AVX or NEON code.

The only major issue, DirectXMath doesn’t support FP64 precision.