There was a cheesy "CSI" video with some Sun engineers building a zfs pool from a crapton of USB sticks. I think I might do that to have a play with it before I spend ~$1500AUD building a NAS.
I actually did this with 64x32GB eBay special (read: godawful slow) MicroSD's. More to make a joke about my "enterprise flash storage system" as we were in the middle of a bunch of storage vendor trials, and sick of dealing with buzzword sales guys.
It was... slow. But the blinky lights were pretty awesome.
We did a similar experiment to http://macguild.org/raid.html (RAID 5 over USB floppy drives) with a bunch of small slow USB sticks we'd collected between us (~64..256Mb IIRC, using just 60M on each to treat them as same sized devices) and various Linux RAID arrangements. It was a good way to play with the tools in an environment where we can break things artificially, as well as an interesting little play thing. Another good place to practise is in VMs of course. Both options are far enough away from a real physical NAS that any performance tests you do are largely irrelevant and some of the failure modes aren't as easy to properly simulate as you might think, but still a useful learning or confidence building exercise if you are new to working with the relevant tools.
Or you could just use multiple partitions on a single scratch disk (or even a ramdisk). A great way to get familiar with the workflow and tools without going to a lot of effort. This is how I first experimented with ZFS.
You can also create files on disk and add them to pools.
There's actually a trick you can use to create a failed ZFS array (e.g. if you want to create a 5+1 array with only five disks), where you use dd to create a sparse file of the appropriate size, which lets you create a '1 TB' file while only writing 1 byte to disk. Add it to your zpool along with the rest of your disks, then fail it out, remove it, and replace it with a physical disk.
It's a good way of taking a drive with 2 TB of data on it and adding it to a ZFS filesystem which includes that disk while also keeping the data, but without using a separate disk (or copying the data twice).
Crap, I should have done that. Oh well, 4 USB sticks was only $24.
Re:ZFS Gotchas
by mrottenkolber on Monday June 02, 2014 @06:17AM
(#7832711)
I actually run such a USB stick raid setup (using regular linux raid/ext4). Its write-speed is horrible but its really cheap and serves well as my personal vault.
Many of these considerations don't have anything to do ZFS per se, but come up in designing any non-trivial storage system. These include most of the comments about IOPS capacity, the comments about characterizing your workload to understand if it would benefit from separate intent logs devices and SSD read caches, the notes about quality hardware components (like ECC RAM), and most of the notes about pool design, which come largely from the physics of disks and how most RAID-like storage systems use them.
Several of the points are just notes about good things that you could ignore if you want to, but are also easy to understand, like "compression is good." "Snapshots are not backups" is true, but that's missing the point that constant-time snapshots are incredibly useful even if they don't also solve the backup problem.
Many of the caveats are highly configuration specific: hot spares are good choices in many configurations; the 128GB DRAM limit is completely bogus in my experience; and the "zfs destroy" problem has been largely fixed for a long time now.