Let me clarify: not performant in any way that we want to implement. The NetApp referenced in that analysis would cost as much as we've spent on both of our OpenStack and Kube clusters. NetApp is great if you're a hospital or a bank but not an internet company.

We need something built on common PC chassis, either as distributed local storage or some type of high speed interconnect.

Storage is definitely one of those businesses where "fast, good, cheap: pick any two" has always applied, and still does.

and as an industry, we are tending towards fast, cheap, and you can write code to work around not having "good"

In my experience, if it isn't good relatively quickly, it certainly doesn't remain cheap for long.

It's not so much about the code as having a usage that can accommodate being run on "not good".

Banks and hospitals don't.

I have lots of issue to get NFS works on some particular db as well. Such as RethinkDB and InfluxDB...

Last I checked, databases came with numerous warning in the docs "don't run me on a network drive".

Is there any reason why you are trying this at all?

Depends entirely on the database as to if they work and why. Oracle, for instance, has a native NFS client built into the database. Modern versions of MS SQL support running on SMB3. Postgres is fine with NFS as well.

The biggest issue people have with NFS is when they roll their own. To get performance they allow asynchronous writes | but this means the write cache can potentially be lost during a power failure. Enterprise NAS systems like the NetApp referenced above have battery backed cache so writes are never acknowledged until they're in a secure medium.

I think that's a good rule of thumb but you can certainly do it if you know what you're doing (which includes having the right hardware).

I don't know how to solve problem of stateful container and data storage so I tried to use a NFS for this and failed.

Not to beat the NetApp horse to death, but they've solved that problem (they just do a poor job of advertising it):

https://github.com/NetApp/netappdvp

I've had many problems with NFS, but they were not performance related. Mostly FS hangs with some program waiting for IO that never comes.

This is the biggest issue I've had with NFS. If the mount goes stale, programs can get stuck in an uninterruptable state that takes a reboot to clear out.

Just mount the file system with "soft,intr" options and you won't have that problem. Otherwise NFS is (perhaps unrealistically) optimistic that the server will come back.