One solution I'm looking at is essentially mirrored raid5 over iscsi but it gets around your raid card single point of failure; the idea is that I have 2 zfs boxes (to begin/for testing these are OpenSolaris xen guests that control the spare drive slots I have in boxes that are primarily doing other things.) on each of those, i zraid the available drives. I then export iscsi luns of some standard size. The client box takes an iscsi lun from each opensolaris box and mirrors them with whatever software raid stack the client likes.

The big win here is caching... each of the opensolaris boxes can be using write-back caching, because all data is mirrored to the other opensolaris box. much like a NetAPP or EMC with dual heads, I don't have to worry about disk inconsistencies caused by write-back cache unless both drives fail. (of course, your 'real' dual-headed SAN will switch to write-through caching if one head dies, and doesn't require a full rebuild when the other head comes back online, but I can't afford a 'real' dual-headed SAN.)

Rebuild times, I imagine, will be quite significant.

Now, I am very leery of the performance of this system (being as these are xen instances, it wouldn't surprise me if it was unsuitable for anything but tape replacement) but I haven't tried yet. It seems, though that it would work just fine if I was not too cheap to use real hardware.

If you wanted to add complexity, you could have 3 or more of these opensolaris boxes, and use software raid5 on the client to save space.

Are your target boxes virtual machines?

I always prefer the strategy where mirroring the data happens before the disk is presented to the VM. Have a look at GlusterFS and see if that might be an option? I know it wasn't playing nice with zfs for a while there, though.

Virtual private servers, yes. I use Xen.

And I agree, giving the user 2 block devices and expecting them to mirror is a very bad idea. I know this from experience. in an earlier setup, I'd give each user two block devices, one from each disk in the box. the idea being the user would mirror or not as appropriate. well, you loose a customers data, you loose the customer. Lesson learned.

But this is where my virtual setup gives me an edge. On every Xen server, I have a control guest or driver domain, the dom0. all disk and net I/O goes through the dom0 anyhow, so there wouldn't be much more overhead to me doing the mirroring on the Dom0, and passing the md through to the DomU (what the user controls.) I'm not adding another point of failure, either; if the dom0 chokes, all guests on the system will crash regardless.