If you cannot go lower, I'd consider a partnership with an existing brand. You'll have to have an insane marketing budget to compete with existing clothing brands without viral.
If you can partner with Brooks Brothers (established in 1818, and part of American preppy folklore), J. Crew, Men's Warehouse, or similar, you'll have an easy time breaking in, although a much harder time getting most of the profit share.
Word-of-mouth can work well, but you've got to understand your customers, the networks, and the levers you can tweak to change rate-of-growth. I was suggesting price/quality/margin as one of those levers -- you want to be appealing to as many customers within your customers' networks -- but there may be others if that one is unappealing.
And obviously we'd go even lower if we could.