You don't need the fab, you hire one of the existing ones to build it for you. nVidia does this for example (and ATI did as well)

You can license the ARM IP from a company as well (I mean, not just the instruction set IP but the actual implementation as well)