News for hackers. Stuff that matters.
Re:Why doesn't `kill -9` always work? by agwa on Sunday January 20, 2013 @08:15PM (#5089207)
Do not mount NFS with "soft" unless you really know what you're doing. NFS' behavior to hang is not "stupidity" - it's actually one of the best things about NFS. Applications do not deal well with failed reads/writes. If there's a brief network interruption or the NFS server goes down, it's WAY safer to cause applications to hang until the server comes back. Since NFS is a stateless protocol, when the server comes back, I/O resumes as if nothing ever happened. This helps make using NFS feel more like using a local filesystem. Otherwise it becomes a very leaky abstraction.

Not being able to kill processes stuck on NFS I/O is annoying though, so you can mount with the "intr" option and that makes such processes killable. However, since Linux 2.6.25, you don't even need this and SIGKILL can always kill applications stuck in NFS I/O.

Re:Why doesn't `kill -9` always work? by dredmorbius on Sunday January 20, 2013 @10:24PM (#5089678)
Expecting NFS (or any other remote filesystem) to behave as if it were local is a fundamental error.

Time, and speed of light, ultimately matter. If you need assurance, find a way of getting reliability in your system through redundancy and locality. Distinguish between "task has been delegated" and "task has been confirmed completed". Down any other path runs pain, and anyone who tells you otherwise is selling something.

You're going to have to compromise: whole systems (or clusters) going titsup because your NFS heads had a fart, or lost commits. Neither is very attractive when shit's on the line.

Your database relying on NFS is a fundamental error you'll have to design around.

Re:Why doesn't `kill -9` always work? by jrockway on Monday January 21, 2013 @01:04AM (#5090166)
NFS can be faster than local storage. Compare a very slow tape drive versus a very fast 10Gbps network connection to an NFS server with a huge in-memory cache.

The problem is expecting any filesystem to be reliable.

Re:Why doesn't `kill -9` always work? by laumars on Monday January 21, 2013 @07:51AM (#5091225)

    > Expecting NFS (or any other remote filesystem) to behave as if it were local is a fundamental error.
Any storage medium is subject to fail, the issue isn't specific to remote file systems. In fact I personally consider the Plan 9 methodology towards file systems to be one of the greatest ideas it's imparted on Linux and Unix. So I'm all in favour of the local file systems, services and remote objects and file systems transparently behaving as one.

     > Time, and speed of light, ultimately matter. 
People often cite the speed of light when talking about electronics when actually electrons don't travel at the speed of light (they have mass). The difference maybe small, but using precise scientific terms imprecisely is one of my pet hates.

    > You're going to have to compromise: whole systems (or clusters) going titsup because your NFS heads had a fart, or lost commits. Neither is very attractive when shit's on the line.
If this is a serious issue then you should be looking into iSCSI. A bad workman blames his tools, a good workman finds the best tool for the job.
Re:Why doesn't `kill -9` always work? by SEMW on Monday January 21, 2013 @08:26AM (#5091348)
> People often cite the speed of light when talking about electronics when actually electrons don't travel at the speed of light (they have mass)

Actual electrons move pretty slowly - of the order of millimetres per hour. That speed has nothing to do with the speed the signal propagates down the wire.

The speed the signal is propagated is the speed the electromagnetic wavefront moves along it. Which is mostly limited by the dielectric constant of the wire's insulator. (That actually is related to the speed of light in the insulator - they both depend on its permittivity). It's not related to the speed electrons move in the wire, which depends on its cross-sectional area and the current. In particular, the fact that electrons have mass isn't relevant to the wavefront propagation speed.

(C.f. fibre optic cables, which will have a wavefront propagation speed not dissimilar to copper wire (i.e. both will be a substantial fraction of c), even though, unlike copper wire, their carriers are massless and actually do move at the wavefront speed).

Analogy: imagine pushing the end of a very long, rigid broomstick. The actual wood in the broomstick moves pretty slowly (maybe you move it a cm in a second). But the person at the other end feels their end of the broomstick move almost immediately, limited only by a speed of light delay (or a little more if the broomstick isn't as rigid as possible).

Re:Why doesn't `kill -9` always work? by yebyen on Monday January 21, 2013 @09:33AM (#5091608)
Damn, you beat me to it. I think you did a better job explaining than I would have, too. An upvote for you, sir!

For any skeptics or laymen who might have been lost at 'electromagnetic wavefront', or who are wondering why electrons would seem to move so slowly when we think of electricity (and even free electrons) as moving so fast, the analogy of the broomstick really does not do it justice, since you can't see the broomstick oscillating very quickly as you might have guessed the electrons must really do to satisfy Heisenberg:

http://en.wikipedia.org/wiki/Particle_in_a_box

http://www.physicsforums.com/showthread.php?t=121456

The keyword is "electron drift velocity"

The important concept to understand is that information does not move trapped inside of individual electrons, and thus is not at all limited by electron drift velocity.

Re:Why doesn't `kill -9` always work? by SummonJetTruck on Monday January 21, 2013 @10:20AM (#5091824)
With a broomstick, wouldn't it be the speed of sound, since it's a pressure wave?
Re:Why doesn't `kill -9` always work? by SEMW on Monday January 21, 2013 @11:09AM (#5092035)
Fair point. For the analogy I was going for a hypothetical rigid broomstick, made of stuff that's as incompressible as physically possible. The speed of sound in that would be c. Yeah, that isn't very realistic, but I don't think that hurts the point it was illustrating.
Re:Why doesn't `kill -9` always work? by laumars on Monday January 21, 2013 @08:51AM (#5091426)
Interesting stuff. Thanks for the correction :)
Re:Why doesn't `kill -9` always work? by dredmorbius on Monday January 21, 2013 @02:43PM (#5093282)
My experience with NFS failures is that they're an order or two of magnitude higher (and much more binary) than local storage issues.

Speed of light matters in that it sets an lower absolute limit on the latency of connections between two points, particularly if they're outside of a single datacenter. While non-local storage can be faster (and more reliable) than local storage within a well-architected datacenter environment, even a link of a few tens of miles, let alone thousands, will introduce SOL latencies, carrier inconsistencies, and switching and other equipment idiosyncrasies. Your point that actual propagation rarely hits SOL speeds only compounds the problem.

I've looked at iSCSI. NFS, for its warts, allows multiple concurrent access, iSCSI binds storage to a single system. Management/diagnosis tools, vendor sophistication, documentation, and support are sorely lacking (or were on my last exposure). NFS may suck, but we can quantify, analyze, diagnose, and understand its suckage.

My own preference is for distributed network/application based storage. Caching for repeated static queries. Direct remote database access (if your network and remote storage are fast enough for NFS, they're fast enough to put the DB server directly on hot storage), Hadoop/MapReduce type solutions for higher-speed access, git or other change-and-push state for less aggressive changes. Getting to my earlier point: this requires architecting your application to take account of remote storage capabilities and limitations. It need not be specific to a given technology (and ideally you'd abstract out to the level of "a cache", "a database", "a keypair storage", "a versioning system"), but you're also not just expecting local filesystem semantics and reliability.

Ironically, gitfs is a thing.

Re:Why doesn't `kill -9` always work? by laumars on Monday January 21, 2013 @06:00PM (#5094267)

    > My experience with NFS failures is that they're an order or two of magnitude higher (and much more binary) than local storage issues.
Usually that's the fault of lower down in the chain than with NFS itself. If you're building a home brew SAN using NFS, then there's a lot of kernel tweaks and such like that you can implement. The Linux kernel is a great all-rounder, but if you're building your own enterprise-grade appliances then there's a lot more work required than just vi'ing /etc/exports.

But anyone using NFS to run databases or for caches needs their head examining. It's purely a protocol for file system sharing, caching should be ramdisk, databases should be using dedicated TCP/IP channels preferred by the RMDBS in question and so on. Using NFS as a blanket solution like that is a little like running websites over NFS and wondering why PHP isn't compiling properly. In fact some versioning systems do use HTTP as their method for syncing.

Re:Why doesn't `kill -9` always work? by dredmorbius on Monday January 21, 2013 @06:36PM (#5094448)
I'm referring to an enterprise implementation comprising over 5000 nodes and somewhere in the neighborhood of 300-400 NFS mounts under typical use (~60-70 at boot, additional mounts via autofs under typical workloads).

Issues: ad-hocery, former Sun shop, growth-through-acquisition (including acquiring the application infrastructure of said firms), 20 years of legacy, much staff attrition (no more than normal, but even 5-10% means virtually complete turnover within that period), geographically distributed network (3-4 continents), etc.

Pretty much a worst pathological case, except that it's very standard in many, many established firms.

Re:Why doesn't `kill -9` always work? by laumars on Tuesday January 22, 2013 @02:07AM (#5095979)
I see.

Will some kind of distributed file system work here? If you can clustered your NFS server you'd (hopefully) end up with more localised nodes for your client to connect to (a little bit like how a CDN works). For what it's worth, I wouldn't even run 5000 simultaneous HTTP requests on a single node in any of my web farms, let alone on a single NFS server.

Also have you looked into kernel level tweaks? I'm guessing you're either running Solaris (being a former Sun shop) and while I am a sysadmin for Solaris, I've not needed to get this low level before, but certainly on Linux, there's a lot of optimisations that can be made to the TCP/IP stack that would improve the performance in this specific scenario.

I do agree with you that 400->5000 active NFS connections does push the realm of practical use, but I don't think that dismisses NFS entirely; it still outperforms all other network mounted file systems.

Re:Why doesn't `kill -9` always work? by smutticus on Monday January 21, 2013 @04:20AM (#5090660)
In my experience NFS is fine as long as the traffic does not need to traverse a WAN link. The problem, as I see it, is that once something is available via NFS everyone wants to mount it from everywhere. NFS simply does not deal well with high RTT connections.
Re:Why doesn't `kill -9` always work? by dredmorbius on Monday January 21, 2013 @06:37PM (#5094449)
Good luck with that assumption.

/net

Re:Why doesn't `kill -9` always work? by NelsonMinar on Sunday January 20, 2013 @09:12PM (#5089437)
This would be a good time to re-read Waldo's "A Note on Distributed Computing", which points out how remote filesystems will never act like local filesystems. http://labs.oracle.com/techrep/1994/smli_tr-94-29.pdf