But honestly, over the last years i have a strong feeling for the old, fast, stable (and most importantly simple!) filesystems like ext* or xfs.
Just yesterday i had btrfs telling me that my disk is full (when in fact it is half full with nearly 400GB free!). Also have seen some major, hard to resolve problems with ZFS years ago.
It led me to believe, why the hell we need those ultracomplicated can-do-everything filesystems?! The only thing i actually care for is fast access without some complicated compression, deduplication and what-not steps in between and data safety. And as far as i can tell, ext4 and XFS are pretty safe. Never seen those destroy files. Ever.
On the other hand finding problems and debugging filesystems in kernel can be a major headache. I want my filesystem to as simple as possible, nowadays. ZFS and btrfs is all nice and so on, but next time i look for a filesystem, i'll go back to the established and simpler ones.
Sadly though, I do agree with your judgement of Btrfs. I trialled it for about 6 months and found it to be conceptually similar to ZFS but very much it's practical polar opposite.
However, like with any software decision, the right file system choice depends as much on the platform's intended purpose and the administrator, as it does on age and features. For file servers, ZFS is an excellent choice; but for low footprint appliances, you'd be better off with ext4 or xfs. And desktops is just a question of personal preference.
For example, i am wondering why i would want to put compression into the file system. Or deduplication. Or the possibility to put some data on SSD and other on HDD. If i have a server and user data, it should be up to the application to do this stuff. That should be more efficient, because the app actually knows how to handle the data correctly and efficiently. It would also be largely independent of the file storage.
I've seen some cases where we had a storage system, pretending to do fancy stuff and fail on it. And debugging such things is a nightmare (talking about storage vendors here, though). But for example, a few years ago we had major performance problems because a ZFS mount was at about 90% of space. It was not easy to pinpoint that. After the fact it's clear, and nowadays googling it would probably give enough hints. But in the end I would very much like that my filesystem does not just slow down depending on how much filled it is. Or how much memory i have.
edit: Also, just to clarify. I think that Sun had one of the best engineers of the whole industry. Everything they did has been great. Honestly, i have huge respect for them and also for ZFS. I still think that ZFS is great, but in the end, i am wondering if it is just too much. For example, nowadays you have a lot of those stateless redundant S3 compatible storage backends. Or use Cassandra, etc. Those already copy your data multiple times. Even if they run on ZFS, you don't gain much. If you run ext4 and it actually loses data, the software cares about that. That's just one case, and of course it depends on your requirements. Just saying, those cases are increasing where the software already cares for keeping the important data safe.
ZFS needs more memory than ext4, but reports for just how much memory ZFS needs is grossly over estimated. At least for desktop usage - file servers are a different matter and thus that's where those figures come from.
To use a practical example, I've ran ZFS + 5 virtual machines on 4GB RAM and not had any issues what-so-ever.
> For example, i am wondering why i would want to put compression into the file system.
A better question would be, why wouldn't you? It happens transparently and causes next to no additional overhead. But you can have that disabled in ZFS (or any other file system) if you really want to.
> Or deduplication.
Deduplication is disabled on ZFS by default. It's actually a pretty nichely used feature despite it's wider reporting.
> Or the possibility to put some data on SSD and other on HDD. If i have a server and user data, it should be up to the application to do this stuff. That should be more efficient, because the app actually knows how to handle the data correctly and efficiently. It would also be largely independent of the file storage.
I don't get your point here. ZFS doesn't behave any differently to ext in that regard. Unless you're talking about SSD cache disks, in which case that's something you have to explicitly set up.
> I've seen some cases where we had a storage system, pretending to do fancy stuff and fail on it. And debugging such things is a nightmare (talking about storage vendors here, though). But for example, a few years ago we had major performance problems because a ZFS mount was at about 90% of space. It was not easy to pinpoint that. After the fact it's clear, and nowadays googling it would probably give enough hints. But in the end I would very much like that my filesystem does not just slow down depending on how much filled it is. Or how much memory i have.
ZFS doesn't slow down if the storage pools are full; the problem you described there sounds more like fragmentation, and that affects all file systems. Also the performance of all file systems is also memory driven (and obviously storage access times). OS's cache files in RAM (this is why some memory reporting tools say Windows or Linux is using GB's RAM even when there's little or no open applications - because they don't exclude cached memory from used memory). This happens with ext4, ZFS, NTFS, xfs and even FAT32. Granted ZFS has a slightly different caching model to the Linux kernel's, but file caching is something that is free memory driven and applies to every file system. This is why file servers are usually speced with lots of RAM - even when running non-ZFS storage pools.
I appreciate that you said none of us are experts on file systems, but it sounds to me like you've based your judgement on a number of anecdotal assumptions; in that the problems you raised are either not part of ZFS's default config, or are limitations present in any and all file systems out there but you just happened upon them in ZFS.
> i am wondering if it is just too much. For example, nowadays you have a lot of those stateless redundant S3 compatible storage backends. Or use Cassandra, etc. Those already copy your data multiple times. Even if they run on ZFS, you don't gain much.
While that's true, you are now comparing Apples to oranges. But in any case, it's not best practice to be running a high performance database on top of ZFS (nor any of CoW file system). So in those instances ext4 or xfs would definitely be a better choice.
FYI, I also wouldn't recommend ZFS for small capacity / portable storage devices nor many real time appliances. But if file storage is your primary concern, then ZFS definitely has a number of advantages over ext4 and xfs which aren't over-complicated nor surplus toys (eg snapshots, CoW journalling, online scrubbing, checksums, datasets, etc).
We don't but some people do. Or think they do, but that's the same. It's not like I'm losing something because I can always use the simpler filesystems myself. I went with ReiserFS for over a decade: never really had to think about my file systems which is what I want, after all. Once you find a good file system, you stop thinking about file systems.
For me, this means two things.
I want my file system to be safe and transactional: either my change gets in or it doesn't, but I don't want to find my file system in some degraded in-between state. Ever. I'm willing to pay for that with cpu time or i/o speed, that's like the first 90% of my requirements.
The second criteria is that it's generally lean and doesn't do anything stupid algorithmically. It should support big files, provide relatively fast directory lookups (so that 'find' will run fast), and have some decent way of packing files onto the disk that doesn't defragment the allocations too badly, and ideally do some book-keeping during idle i/o so that I don't really have to run a defragmenter, ever. But these are kind of secondary requirements that aren't worth anything unless the file system keeps my files uncorrupted and accessible first and foremost.
The trick is making the data change, problem detection, and recovery mechanisms simple. Then, each feature is a function that leverages that in a way that's easier to analyze. The features themselves can be implemented in a way that makes their own analysis easier. So on and so forth. Standard practice in rigorous, software engineering. Not quite applied to filesystems...
This is a major ongoing issue with Docker on Btrfs in particular. E.g. CoreOS switched to Overlayfs + Ext4 by default to reduce that problem.
In the 80's-90's, many systems aiming for better robustness or management realized filesystems were too complex. So, they instead implemented storage as objects that were written to disks. Many aspects important for security or reliability were done here on this simple representation. The filesystem was a less privileged component that translated the complexities of file access into procedure calls on the simpler, object storage. Apps that didn't need files per se might also call the object storage directly. Some designs put the object storage management directly on the disk with on-disk, cheap CPU. Supported integrated crypto, defrag, etc. NASD's [1] and IBM System/38 [2] are sophisticated examples in this category.
The other model was building complex filsystems on simpler ones. The clustered filesystems in supercomputing and distributed stores in cloud + academia are good examples of this. The underlying filesystem can be something simple, proven, and highly performing. Then, the more complex one is layered over one or more nodes to provide more while mostly focusing on high-level concerns. Google File System [3] and Sector [4] are good examples.
So, we can have the benefits of simple storage and complex filesystems with few of their problems. That there's many deployed in production should reduce skepticism that it sounds too good to be true. Now, we just need more efforts in these two categories to make things even better. Nice as ZFS and BTFS look, I'd rather they had just improved XFS in directions of these categories instead. Duplicated effort would've led to innovation instead on top of any innovations they produced.
[1] http://www.pdl.cmu.edu/PDL-FTP/NASD/CMU-CS-97-185.pdf
[2] http://homes.cs.washington.edu/~levy/capabook/index.html
Definitely lost way too many files to XFS zeroing things out on a crash.
Turns out supercomputer software might be written to a different set of requirements... Who knew!
EDIT to add: check your hard disks and OS's you plug them into, too. I lost a ton of data once over a bug in a driver that looked like a filesystem bug but had nothing to do with it. Several layers where issues can pop up but aren't obvious from their results.
(Yeah, I know they exist. I've just never been in some place as big as to actually use them.)
I use git annex, which has this built-in; just run "git annex repair" and it'll verify & fix any damaged files.
They're called VIA Artigo's:
http://www.ebay.com/sch/items/?_nkw=via+artigo&_sacat=&_ex_k...
Stumbled onto them looking for accelerated crypto & cheaper x86 boxes. So, 6x Artigo at $300 (back then), 3x UPS at $50 each, 2x network switch's at $50, and 12x cables at $5 each = a very high availability solution for 100GB+ data for $2,110 plus tax. Breaking 1TB just took an extra grand or so. At that time, getting a similar level of storage with only two-way redundancy from "bargain" server vendors was more like $10,000-20,000.
Just gotta invest wisely and your problem turns from "cost too much" to "wow I'm sinking a lot of time into this project." The latter is more fun, at least. :)
Then there was my evolved scheme of using diversity where I had different OS's or filesystems with stuff handled at application layer. More complex to set up but decreases chance any one set of code is going to mess you up. Safety or security through diversity is a powerful technique. Harder to do today with so much code reuse: things might only be different on the surface with same bugs lurking underneath. All these different web security schemes using the same OpenSSL library is a good example.
I still use reiserfs when I have to run linux (these days I'm mostly FreeBSD with ZFS).
http://www.xfs.org/index.php/XFS_FAQ#Q:_Why_do_I_see_binary_...
Also, there's xfs_repair(8).
That said, this does seem to be a big problem for anyone depending on XFS by itself. Sounds like filesystem developers should dig in and fix that zero's problem among any other that are known. Just seems wasteful to ditch such a good, battle-hardened filesystem over a few bugs. Better to just fix them.
With the advance in SSD, HDD will be mainly used for big files, so XFS may yet end up dominating.
I don't know if ext4 is equally good on the SSD or not.
"With the advance in SSD, HDD will be mainly used for big files, so XFS may yet end up dominating."
Good point. We're already seeing that with all the videos and games. So, there might be a resurgence of XFS use. I doubt it will dominate, though, unless it stays the default in enough distro's.
Details please! What are they and what did you do?
https://en.wikipedia.org/wiki/XFS#Performance_considerations
(And disadvantages section.)
I mostly used lowest common denominator of it with backups, clusters, and so on. Any other issues I countered with storage design and recovery procedures. I often did recovery around once a week or month to prevent silent buildup of issues. So, there might be issues I'm not aware of but it worked fine almost every time.
Another commenter mentioned audio skips which happened to maybe a handful of my own files out of GB of music. Could be one of those issues or a quirk with that commenter's setup.
Still my favorite filesystem for what was my favorite supercomputing & NUMA company.