> My experience with NFS failures is that they're an order or two of magnitude higher (and much more binary) than local storage issues.
Usually that's the fault of lower down in the chain than with NFS itself. If you're building a home brew SAN using NFS, then there's a lot of kernel tweaks and such like that you can implement. The Linux kernel is a great all-rounder, but if you're building your own enterprise-grade appliances then there's a lot more work required than just vi'ing /etc/exports.But anyone using NFS to run databases or for caches needs their head examining. It's purely a protocol for file system sharing, caching should be ramdisk, databases should be using dedicated TCP/IP channels preferred by the RMDBS in question and so on. Using NFS as a blanket solution like that is a little like running websites over NFS and wondering why PHP isn't compiling properly. In fact some versioning systems do use HTTP as their method for syncing.
Issues: ad-hocery, former Sun shop, growth-through-acquisition (including acquiring the application infrastructure of said firms), 20 years of legacy, much staff attrition (no more than normal, but even 5-10% means virtually complete turnover within that period), geographically distributed network (3-4 continents), etc.
Pretty much a worst pathological case, except that it's very standard in many, many established firms.
Will some kind of distributed file system work here? If you can clustered your NFS server you'd (hopefully) end up with more localised nodes for your client to connect to (a little bit like how a CDN works). For what it's worth, I wouldn't even run 5000 simultaneous HTTP requests on a single node in any of my web farms, let alone on a single NFS server.
Also have you looked into kernel level tweaks? I'm guessing you're either running Solaris (being a former Sun shop) and while I am a sysadmin for Solaris, I've not needed to get this low level before, but certainly on Linux, there's a lot of optimisations that can be made to the TCP/IP stack that would improve the performance in this specific scenario.
I do agree with you that 400->5000 active NFS connections does push the realm of practical use, but I don't think that dismisses NFS entirely; it still outperforms all other network mounted file systems.
Speed of light matters in that it sets an lower absolute limit on the latency of connections between two points, particularly if they're outside of a single datacenter. While non-local storage can be faster (and more reliable) than local storage within a well-architected datacenter environment, even a link of a few tens of miles, let alone thousands, will introduce SOL latencies, carrier inconsistencies, and switching and other equipment idiosyncrasies. Your point that actual propagation rarely hits SOL speeds only compounds the problem.
I've looked at iSCSI. NFS, for its warts, allows multiple concurrent access, iSCSI binds storage to a single system. Management/diagnosis tools, vendor sophistication, documentation, and support are sorely lacking (or were on my last exposure). NFS may suck, but we can quantify, analyze, diagnose, and understand its suckage.
My own preference is for distributed network/application based storage. Caching for repeated static queries. Direct remote database access (if your network and remote storage are fast enough for NFS, they're fast enough to put the DB server directly on hot storage), Hadoop/MapReduce type solutions for higher-speed access, git or other change-and-push state for less aggressive changes. Getting to my earlier point: this requires architecting your application to take account of remote storage capabilities and limitations. It need not be specific to a given technology (and ideally you'd abstract out to the level of "a cache", "a database", "a keypair storage", "a versioning system"), but you're also not just expecting local filesystem semantics and reliability.
Ironically, gitfs is a thing.