As long as the ARM-community is fragmented, their research/investments won't really be as aligned as Xeon and/or EPYC servers.
HiFive / RISC-V aren't anywhere close to the server-tier.
Why does this matter? If popular OS distributions consistently target ARM-based CPUs, with a sufficient number of packages (esp. development-support-related) working on them, then who cares about fragmentation? An organization could buy systems with ARM chips and software will basically "just work".
Same argument for consumer PCs, although there you have the MS Windows issue I guess.
The more fragmented your community, the harder it is for software to work consistently across all of them. Intel vs AMD has plenty of obscure issues (see "rr" project, and all the issues getting that debugging tool to work on AMD even though it has the same instruction set).
Sound, WiFi, Ethernet, southbridges, northbridges, PCIe roots. You know, standard compatibility issues that having a ton of SKUs just naturally makes more difficult. Having a "line" of southbridges / consistent motherboards does wonders for compatibility (fix the BIOS/UEFI bug in one motherboard, fix it for all) in Intel/AMD world.
But just as AMD has AMD-specific motherboard bugs, and Intel has Intel-specific motherboard bugs... I'd expect Graviton to have its share of bugs that are inconsistent with Apple M1 or Ampere Altra.
Graviton is a standard N1 neoverse core, which is slightly slower than a Skylake Xeon / Zen2 EPYC. There's hope that N2 will be faster, but even if it is, we don't really have an apples-to-apples comparison available (since Amazon doesn't sell that chip).
The most likely source of Neoverse cores is the Ampere Altra, which is expected to have N2 cores shipping eventually. As usual though: since Ampere has lower shipping volume than other companies, the motherboards are very expensive.
x86 (both Intel and AMD) have extremely high volumes: so from a TCO perspective, its hard to beat them, especially when you consider motherboard prices into the mix.