Reality, as always, is not that simple. Matrix analysis is a huge area in itself; and matrices can also be used to represent tensors (which generally are not seen as linear maps) and some other things.