Most brand's factory stores use different production factories or textiles than the non-factory stores. J Crew Factory and Brooks Brothers Factory stores are mostly garbage produced by the same designers but at much lower cost than their mall stores. I would recommend avoiding them unless you're looking for something disposable.

Source: my wife works in textile sourcing