I think that deploying onto a shared volume is exactly the workload that EFS encourages. That things will stop functioning can be said about any (AWS) technology. "Lord help you when (S3|RDS|SQS|etc.) goes down and every (thing) you have is hung up on a broken (thing) connection."

The rational argument here is that if uptime is important to you, the solution is to utilize multiple regions, of which EFS is now in 4. You are using AWS because of the SLA and the assumption that when something goes wrong there are legions of technical folks trying to fix it as soon as possible.

The difference is that all of those services are accessed via an API, which means your client can do clever things when it fails, like timeout, give a fallback, find an equivalent resource in another zone or region, etc.

With EFS, it's exposed as NFS, which hooks in much deeper. If it goes down, there isn't anything you can do to work around the problem, unless you start hacking your own file system kernel modules.

Can't you measure and timeout on the file-descriptor writes too?

Do you want to start wrapping your system calls to read files in code that forks to periodically check for a hung system call?

It's worse that that, actually. With hard mounts (the default), the only way to interrupt is with SIGKILL, which if my memory serves me correctly, is a process-wide signal. So you'd have to do all NFS I/O in a separate child process.

With soft mounts (does EFS support this?) you can exchange SIGKILL for religiously checking return values for all I/O syscalls, retrying partial writes and whatnot.

Can't you just use soft mounts? Especially since in the scenario you described you can use read only mounts.

It depends on the nature of the failure. If you're doing something which triggers hard failures – i.e. where a read() call on a ocket would unblock with an error – soft mounts will eventually recover. The problem is that, in my experience anyways, the vast majority of failures aren't that clean – things like a server which processes packets but never responds, a network connection which drops packets but doesn't change the link status, etc. — and in those cases soft mounts behave no better than hard mounts. There also used to be kernel bugs in *BSD, Darwin, and Linux where the client could deadlock in heavy activity, which were hard to reproduce and get fixed.

In all of those cases, anything which tries to access something on the NFS mount will block in the kernel (i.e. “kill -9“ won't work) and the mount cannot be unmounted normally.

I wrote https://github.com/acdha/mountstatus awhile back – if memory serves, 2004 or so – because we found that on Linux a lazy unmount would still work in this case and so you could have a process monitor the mount status (fork() a child to check the mount, alert if it doesn't get a response within a set interval) and a watchdog could respond to an alert by issuing a “umount -l” and remounting, which doesn't fix the blocked process but is less disruptive than rebooting and that new processes won't block because they tried to access that mount.

Assuming all assets get stored on S3, the other common writing scenario is logging. How would multiple machines writing to the same log file work?

Generally, you wouldn't write to the same log file. Prepending/Appending the instance name to the file would be enough to make it unique.

It wouldn't. NFS is not fully POSIX compliant, largely due to not being cache coherent. In particular, O_APPEND is not supported.

Isn't the whole point of AWS that you can blame AWS engineers for downtime instead of your own (in this case) code deployment thingy?

I'm not sure if you're being sarcastic or not, but no, it isn't. It's to provide building blocks for infrastructure. Down time is still on you. Only a lazy engineer blames their tools for their failures.

    > Down time is still on you

If you can say "It's not our fault, it's Amazon's", there are plenty of boards and customers who are fine with that, in my experience.

Yep. "Nobody ever got fired for buying IBM" applies here.

Not sarcastic: aws is not a tool, it's a service. You're exchanging dollars for SLA. The SLA is built in: you're sometimes (rarely) down, and someone else gets called about it.

Your customers will blame you: you don't want it to happen. Even if amazon pay you some damage you may lose trust or your whole business in the meantime.

With a major AWS outage, customers will think "the Internet is having trouble today." Half the sites they visit will be down or broken in some way.

I think this perspective really depends on who you are supporting. This is my experience:

1) Actual customers: They want you to fix it. They don't care who's fault it is, they are dependent on you.

2) Internal customers: They want someone to blame. They don't really care that it's down, they just want to point the finger at someone when their report goes out late.

"Down time is still on you" - well put. And it is actually great when you can only blame yourself because then you can fix the issue right away. If someone other is to blame, you need to a) convince that person that there is a problem and b) wait for the issue to rise to the top of their queue. Which can be painful.

But in the case of Amazon they will likely know about the problem before you would have anyway. Sure, their problems tend to have a much larger scale, but their ops teams are pretty good.

Just because it's your fault, that doesn't mean that you can quickly and easily fix the problem yourself. Amazon has a team of people to keep EFS up and running, and AWS owns the entire storage, server, and network stack. When your own NFS server starts timing out, you may have only a few people to diagnose the problem.

Do you think that's still true in a "serverless" architecture?

Surprise! Serverless runs on... gasp servers!

Hrm, so for the site 'whoownsMYavailability', the answer is someone else?

I have to admit that is the first use case that I thought of when I read this. No more messing around with AMIs and launch config whenever an update is deployed.

You bring up a good point on the EFS being a point of failure though. What use case do you think the EFS is good for? It seems like even worse for data storage since IO can easily become a bottleneck.

> What use case do you think the EFS is good for

Out of band big data/data pipeline processing. Something where you want an easy way to sync your data but can handle extended downtimes.

Or scenarios where you can use EFS to stage data 'for something else' (i.e. shared image / upload content that stages to a CDN, etc.)

Keep in mind (and this comes from hard experience in 'traditional' NFS web server architecture) - if you mount everything on an NFS volume, you ensure that

1) If something goes wrong on that NFS mount, everything goes wrong. (bad code deploy? All nodes are down!)

2) If you rely on an NFS mount to store everything (e.g. trust keystores for JVMs,etc.) your entire infrastructure is dependent on the I/O capabilities of that NFS mount.

3) No matter how clever you are (or how much you trust NFS clients/versions) you will deal with file locking if you are doing a fair amount of read/write from multiple nodes to a single NFS mount.

Short story - EFS will make some of the 'hard' things with distributed nodes possible, but don't make the easy things impossible to troubleshoot.

The instance can fetch the upgrade from the NFS mount and also work using the previous version when NFS is down. I see no problem here.

but thats the same with any component. how is that different to SNS et al?

NFS is actually multi homed, so there is no real reason why it can't be HA/clustered, apart from the block store and the underlying file system.

> how is that different to SNS et al?

It's not really, and you should write your software to account for SNS outages, etc.

That being said, this is presented as NFS. NFS has a nasty habit of freezing your system if it breaks. If SNS breaks, you get errors and timeouts in your logs, but can keep going. If NFS breaks, you pretty much just sit around waiting for it to come back.

If people want to deploy that way, I suppose that using rsync to update a local clone, and running from that, would minimize this problem.

Most app / language load everything at startup.