If we are being pedantic, transformers operate on integer vectors, and in the process turn them into high dimensional matrices.