News for hackers. Stuff that matters.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by josefbacik on Sunday May 22, 2016 @12:40PM (#11749232)
Kvm (or any overwrite workload for that matter) is the worst possible workload for btrfs because of COW. We have ideas to address this but honestly it's not high on the list.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by SXX on Sunday May 22, 2016 @12:57PM (#11749296)
This is sad since even with some performance loss compression and offline deduplication for VM images sounds like nice feature.

Is there any docs that explain why COW make Btrfs so slow compared to QCOW2?

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by josefbacik on Sunday May 22, 2016 @01:34PM (#11749443)
There are no docs. I'm not a qcow2 expert, what I know is very basic so anything I say about qcow2 can be very wrong.

So qcow has a read only base image that gets updated when we change things. The image format just had the changes from the original image. So you update a package, it adds some metadata to point at the new stuff and adds the data in and you are done.

So with btrfs you have this image on top of btrfs, so you update a file and its metadata inside the image. Say you start with a pristine image that's in nice big extents. You update a package which changes small chunks all over the file. Let's say you update 12 4K extents. So now instead of one extent you now have 36 extents. This affects everything, fsyncs take longer because there's more extents we have to write out, the space is more fragmented so cold cache reads are more expensive, the csums are no longer contiguous so they also take up a larger more fragmented area. It has this really terrible cascading effect.

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by ryao on Sunday May 22, 2016 @02:47PM (#11749719)
That is a problem with CoW in general. ZFS uses a variable level indirect block tree. That puts an upper bound on fragmentation.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by SXX on Sunday May 22, 2016 @01:43PM (#11749480)
I tested it with preallocated raw images on Btrfs and that setup was slow as well. If I get you right it's must be possible to somehow allign VM disk sectors to Btrfs sectors for performance boost. Or compression and deduplication usage going to kill performance anyway?
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by cmurf on Sunday May 22, 2016 @04:42PM (#11750202)
Btrfs internally use logical addresses for all extents. The making from logical to physical is done via the chunk tree which not only indicates physical sector but also the device. So the reference for a file extent says nothing about what device the extent is on or the replication (raid profile) since that is all a function of the chunk and dev trees.

I think Btrfs for a guest F's is best pointed to an LV, rather than qcow2. It's been awhile since I benchmarked that compared to 'qemu-img create -f qcow2 -o nocow=on' which will set xattr +C on the file making it nocow. The nocow xattr helps a lot with this problem.

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by ryao on Sunday May 22, 2016 @05:18PM (#11750326)
ZFS also does better when the guests are stored on zvols. If volblocksize=4K is set on creation, it can avoid read-modify-write overhead. Regular files can be used too, but those are somewhat higher overhead. In that case, recordsize=4K can be used.

That said, I do not think the tests involved nesting CoW file systems.

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by loeg on Sunday May 22, 2016 @01:39PM (#11749468)
Shouldn't ZFS have similar problems with overwrite workloads?
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by ryao on Sunday May 22, 2016 @02:42PM (#11749697)
> Shouldn't ZFS have similar problems with overwrite workloads?

ZFS does suffer from read-modify-write on partial record writes. The effect of that is apparent in the benchmarks. However, the benchmarks are being done on mechanical disks, which have low IOPS. The IOPS of a mechanical disk are roughly the same on a given sequence of IOs at different positions regardless of whether they are 4KB or 128KB in size, so it only has to pay a penalty once. If the record size were changed to 4KB, this penalty would disappear and ZFS performance should increase, provided that the VM internals are properly aligned.

Also, read-modify-write overhead reduces IOPS by at most 2 and bandwidth to the smaller of the link bandwidth and the IOPS times the record size. A CoW filesystem should be able to perform roughly at that level when it does read-modify-write on records/extents. Unless btrfs' internal extents are huge, there is an issue somewhere. Of course, having huge extents by default on which read-modify-write is done could also be considered a design issue.

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by josefbacik on Sunday May 22, 2016 @01:43PM (#11749477)
ZFS has a 7 year head start on BTRFS and has traditionally had about 5 times as many core contributors at any given time so I imagine they've solved this in some novel way by now.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by qaq on Sunday May 22, 2016 @04:58PM (#11750265)
It's not just years it's the types of deployments. ZFS backs a very large number of very large critical systems.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by tolle on Sunday May 22, 2016 @03:26PM (#11749878)
Wasn't ZFS even "production ready" before BTRFS development even began?

I don't remember the state of it when it was first introduced into Solaris, maybe someones memory is better then mine. Was ZFS better of in 06-07 Then BTRFS is now?

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by ryao on Sunday May 22, 2016 @03:32PM (#11749896)
btrfs development started in 2007 while ZFS development started in 2001. If you want to do a point in time comparison, look at how ZFS was in 2010. You can get the last copy of OpenSolaris for the comparison. It would need to be done on physical hardware due to poor support for virtualization back then.

By the way, ZFS was deemed production ready after 4 years of development.

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by tete on Sunday May 22, 2016 @04:10PM (#11750064)
That's true. One quick side note though: BTRFS actually had a fairly long "draft period" (multiple years) of design. I have no clue how long that was on ZFS. I just know that the original author mentioned that somewhere.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by ryao on Sunday May 22, 2016 @04:20PM (#11750100)
If you have time, you can watch this video where Jeff Bonwick describes the birth of ZFS:

http://m.youtube.com/watch?v=dcV2PaMTAJ4

I am the guy who asked the LZJB question. To summarize my recollection, formal design work on ZFS started in 2001 when Matthew Ahrens started working at Sun. Jeff Bonwick had promised Matthew Ahrens a job at Sun making a filesystem a 6 months to a year before then when Matt was still in college. I am sure that both Jeff and Matt had some thoughts on it during that time, but there was no formal effort until Matt's employment started.

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by ryao on Sunday May 22, 2016 @03:43PM (#11749944)
The idea that btrfs has fewer contributors ought to be a surprise for many. Users tend to assume the opposite. ZFS having 5x more seems to be on the high side to me though. How did you determine that?

By the way, I was under the impression that ZFS development started in 2001 while btrfs development started in 2007. That would be a 6 year difference.

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by rurban on Sunday May 22, 2016 @05:31PM (#11750372)
5x more core ZFS contributors sounds to me worse and not better than BTRFS. The more the worse usually.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by josefbacik on Sunday May 22, 2016 @06:59PM (#11750652)
Not sure how that tracks. These file systems are huge code bases with lots of problems and areas for improvement. When you only have 2-4 people who understand most of it at a given time things move slowly. If I could tell somebody to go fix balance my life would be a whole lot easier, but in reality we end up being interrupt driven and have to prioritize differently.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by biokoda on Sunday May 22, 2016 @12:44PM (#11749244)
What kinds of workloads is it designed for then?
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by josefbacik on Sunday May 22, 2016 @12:55PM (#11749288)
It's a general purpose file system, it does great with metadata heavy workloads and normal streaming writes. The overwrite case is special because of COW. Eventually we'll be comparable to everybody else but we aren't now.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by avar on Sunday May 22, 2016 @02:57PM (#11749772)
I reinstalled my laptop with btrfs on / because "why not" about a year ago.

I ran into issues with maintaining a huge ~1M file Maildir, it seemed to do very badly with huge directories. Some kernel thread would be at 99% CPU while I was trying to populate a Maildir, and the entire system ground to a halt.

More importantly I would run into issues like running out of space on / with 20G left, but it was the "wrong kind of space". I.e. I had run out of metadata space but had plenty of file space left and had to run a rebalancing operation on the filesystem.

I didn't need any of the COW etc. advanced features that btrfs provides, and didn't test them, but as just a normal user needing a general purpose filesystem it was a bit too much hassle, especially with needing to administer the filesystem in a manner similar to how I might administer a RDBMS.

I ended up just going back to ext4.

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by tadfisher on Sunday May 22, 2016 @03:10PM (#11749821)
A subvolume for your Maildir, mounted with "-o nodatacow", would have prevented the first issue.

If btrfs could modify that behavior automatically using heuristics it would truly be general-purpose. However there are some inherent tradeoffs of that approach that users need to consider, and I consider that downside a basic constraint of the Unix filesystem abstraction.

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by avar on Sunday May 22, 2016 @03:34PM (#11749908)
I think I did try stuff like that, including mounting / with nodatacow and other similar options.

I walked away with the impression that btrfs was a powerful tool for certain use-cases, but definitely not something I'd call a "general purpose" filesystem given the need to be fairly knowledgeable about its internals to use it for common desktop use-cases.

To me a general purpose filesystem is something like ext4, it doesn't have amazing performance, it's not bad either, but generally nothing unexpected happens with it and I can just leave it there and don't have to worry about it.

The btrfs filesystem seemed like the opposite of that. A very powerful tool whose power and flexibility made it less general purpose by virtue of needing to keep close tabs on how you were using it.

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by ryao on Sunday May 22, 2016 @04:03PM (#11750033)
If you are open to trying again, I suggest trying ZFS. It should work well here.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by ryao on Sunday May 22, 2016 @03:54PM (#11749998)
nodatacow means no checksums. Whatever performance problem it was intended to resolve should also recur the moment that a snapshot is done, because then CoW re-enters the picture.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by rodgerd on Sunday May 22, 2016 @08:05PM (#11750893)
Honestly, after more than two years of running btrfs on data I care about, my answer would be "testing your backups".
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by daurnimator on Sunday May 22, 2016 @01:06PM (#11749335)
But shouldn't NOCOW mitigate that? (which the article does for some tests)
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by josefbacik on Sunday May 22, 2016 @01:36PM (#11749455)
NOCOW is horribly expensive because we still have to go check and make sure that there are no snapshots pointing at the changing extents. It only solves the fragmentation issue, and if you don't prealloc your image it doesn't even do that.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by ryao on Sunday May 22, 2016 @04:02PM (#11750026)
Are you sure that disabling CoW solves btrfs' fragmentation issue when you pre-allocate the image? If the volume is snapshotted regularly, CoW should be active once on each extent once until the next snapshot reactivates it. That should mean that the fragmentation would still occur, although more slowly. Is that correct or am I misunderstanding something?

I am only superficially familiar with btrfs internals, but I do not see any way to implement snapshots without either doing CoW or duplicating the data in its entirety. If you are checking to see if the data is part of a snapshot, then you should be doing CoW.

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by josefbacik on Monday May 23, 2016 @02:20AM (#11752237)
Yeah nocow only works if there are no snapshots, which is why we have to check, so if you use snapshots it falls back to COW.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by MertsA on Sunday May 22, 2016 @06:42PM (#11750604)
Disabling CoW disables snapshotting.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by versteegen on Sunday May 22, 2016 @10:37PM (#11751452)
(I'm not an expert, just a user) Snapshotting effectively disables NOCOW. Snapshotting always works, and needs COW to do so. Which means ryao is right that fragmentation will occur. btrfs doesn't copy a NOCOW file in its entirety if it's in a snapshot, or I would have run out of disk space by now. (I really need to move my VM images to a non-backed-up subvolume.)
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by josefbacik on Monday May 23, 2016 @02:17AM (#11752230)
No it doesn't.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by versteegen on Sunday May 22, 2016 @10:44PM (#11751486)
How about if you have a NOCOW subvolume with just a few files on it, e.g. 3 VM images. Is it still expensive to check for snapshots? Does the cost of checking disappear if you have no snapshots of the volume? As ryao mentioned, you'd still get fragmentation if you have snapshots, so in that case is using NOCOW for a VM image counterproductive?

Seems like the expense of checking could be largely removed with clever enough metadata caching, which probably noone has had time to implement.

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by josefbacik on Monday May 23, 2016 @09:10AM (#11753587)
We have to look up the physical extent in the extent reference tree, so the cost is independent of the number of snapshots and more a function of the fragmentation of the extent tree. The metadata is all cached of course, but fragmentation means you are likely to not find the entries in cache.

The other aspect that I haven't talked about is our fsync performance is kind of shit compared to other fs'es. Now this does get better in the nocow case but it's still pretty heavy and needs optimization.

Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by zbraniecki on Sunday May 22, 2016 @09:59PM (#11751302)
Is brtfs a good choice from the power consumption standpoint? I'm on a laptop and trying to maximize my battery life, and am wondering if my choice of brtfs limits it.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor... by jwatte on Sunday May 22, 2016 @11:54PM (#11751756)
You could try nilfs for that.