And you make a good argument about handling read errors.
In practice that's not what high availability is about though, and it's certainly not what we're talking about with AWS, iSCSI or even an HA-NFS solution.
Availability in those terms is properly designed, redundant infrastructure. There are many cases I'm sure (say a Stock-Exchange) where HA-NFS would not be appropriate. In your average web-app however, request failures while transitioning to failover are (in my experience) just part of the bargain.
In that context, waiting 60 seconds for a stale NFS handle to pick a new connection up on the VRRPed failover node is acceptable, and like you said, you make assumptions and don't worry about the edge-case here.
There are clearly situations where NFS wouldn't work well, but those are definitely the exception to the rule. There are not a lot of reasons/scenarios where iSCSI couldn't or shouldn't replace local storage for virtualization needs however. In my opinion.
iSCSI is simple and extremely reliable. Perhaps even more reliable than local disks, especially if the primary concern is availability.
Frankly, I don't know that these rules are really something that applies at the level of a polished iSCSI or NFS stack. They certainly apply to the internals of those systems. While you or I aren't going to worry about wether a buffered-file-reader can fetch the next 8K block, that thought and engineering effort definitely has been put into these protocols and implementations at a lower level than you or I will typically work with.
In particular, every time you so much as touch the network, you need a code path to handle the possibility that the network communication utterly failed, just disappeared into oblivion. You just don't code that way when you deal with files; the file may be gone, the disk may run out of space, but you don't sit there in the middle of a fast read loop and handle failure on every read, such as a topology change, i.e., "the disk drive is no longer present". That's not because it's not possible, because you can certainly yank out a USB key mid-read, but you tend not to handle it in most userland code. In fact the OS won't even let you handle many failures, as it will either abstract away the retries or simply lock your userspace hard.
So the fundamental idea is that filesystem abstractions built for relatively reliable communication simply isn't correct for network communication.
Incidentally, the same argument can pretty much be made against remote RPC by simply replacing "disk access" with "function call". You do not have to guard every function call you make with a check for absolute failure. (Not the function returning failure or the function throwing an exception, but the function that may have been there two minutes ago unexpectedly being entirely gone.) RPC may look cute in the source code but it's the wrong layer of abstraction.
[1]: http://news.ycombinator.com/item?id=2470865
[2]: http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Comput...