Conjugation only makes sense for square matrices. More generally, left and right multiplication correspond to coordinate transformations of the output and input space, respectively.

If A is n x n and P is n x p then P is not square but P^T A P is well-defined.