But AFAIK ARM hasn't provided any server specific design. Most ARM based server attempts so far have been with custom designs (e.g. AppliedMicro X-Gene).

Give it time, why wouldn't ARM start offering server building blocks? There is obviously a market for it. ARM went to 64 bit, and it has just gone to advanced GPU designs. It supports extreme number of cores now too. Server-centric designs can not be far off.

While ARM Holdings reference processors are reasonably good, the company is specialized in low power designs. Apple for example had to do a custom design to get the high performance ARM it needed. It is not obvious that ARM has the knowledge, the money and even the will to pursue a server design.