I don't know that we do that. For instance, physicists are able to describe motion in physical terms and don't insist that it literally is a vector. It only happens with quantum fields.