Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor...
by daurnimator on Sunday May 22, 2016 @01:06PM
(#11749335)
But shouldn't NOCOW mitigate that? (which the article does for some tests)
[ Parent
| on HN ]
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor...
by josefbacik on Sunday May 22, 2016 @01:36PM
(#11749455)
NOCOW is horribly expensive because we still have to go check and make sure that there are no snapshots pointing at the changing extents. It only solves the fragmentation issue, and if you don't prealloc your image it doesn't even do that.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor...
by ryao on Sunday May 22, 2016 @04:02PM
(#11750026)
Are you sure that disabling CoW solves btrfs' fragmentation issue when you pre-allocate the image? If the volume is snapshotted regularly, CoW should be active once on each extent once until the next snapshot reactivates it. That should mean that the fragmentation would still occur, although more slowly. Is that correct or am I misunderstanding something?
I am only superficially familiar with btrfs internals, but I do not see any way to implement snapshots without either doing CoW or duplicating the data in its entirety. If you are checking to see if the data is part of a snapshot, then you should be doing CoW.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor...
by josefbacik on Monday May 23, 2016 @02:20AM
(#11752237)
Yeah nocow only works if there are no snapshots, which is why we have to check, so if you use snapshots it falls back to COW.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor...
by MertsA on Sunday May 22, 2016 @06:42PM
(#11750604)
Disabling CoW disables snapshotting.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor...
by versteegen on Sunday May 22, 2016 @10:37PM
(#11751452)
(I'm not an expert, just a user) Snapshotting effectively disables NOCOW. Snapshotting always works, and needs COW to do so. Which means ryao is right that fragmentation will occur. btrfs doesn't copy a NOCOW file in its entirety if it's in a snapshot, or I would have run out of disk space by now. (I really need to move my VM images to a non-backed-up subvolume.)
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor...
by josefbacik on Monday May 23, 2016 @02:17AM
(#11752230)
No it doesn't.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor...
by versteegen on Sunday May 22, 2016 @10:44PM
(#11751486)
How about if you have a NOCOW subvolume with just a few files on it, e.g. 3 VM images. Is it still expensive to check for snapshots? Does the cost of checking disappear if you have no snapshots of the volume? As ryao mentioned, you'd still get fragmentation if you have snapshots, so in that case is using NOCOW for a VM image counterproductive?
Seems like the expense of checking could be largely removed with clever enough metadata caching, which probably noone has had time to implement.
Re:ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Perfor...
by josefbacik on Monday May 23, 2016 @09:10AM
(#11753587)
We have to look up the physical extent in the extent reference tree, so the cost is independent of the number of snapshots and more a function of the fragmentation of the extent tree. The metadata is all cached of course, but fragmentation means you are likely to not find the entries in cache.
The other aspect that I haven't talked about is our fsync performance is kind of shit compared to other fs'es. Now this does get better in the nocow case but it's still pretty heavy and needs optimization.