Re:Data Deduplication with Linux
by Dylan16807 on Saturday August 27, 2011 @09:20PM
(#2933013)
The tricky part seems to be 'too big to fit into memory'. From what I understood and calculated the dedup tables on my system should have been well under 100MB, and the amount of memory designated for metadata was over 350MB, yet the performance was terrible.
[ Parent
| on HN ]
Re:Data Deduplication with Linux
by devicenull on Saturday August 27, 2011 @09:53PM
(#2933075)
Based on my testing (not published anywhere, sorry) ZFS dedup works best when you enable compression. With compression, it's only slightly slower then without dedup.
Re:Data Deduplication with Linux
by Dylan16807 on Saturday August 27, 2011 @10:39PM
(#2933115)
I did have compression on. Good to know that in some cases dedup will perform quite well. Was that with an SSD?
My best guess is that I either ruined the configuration in some way or dedup and only dedup reacted horribly to being in a virtual machine.
ZFS is designed to have lots of horsepower and memory thrown at it.......big servers, available CPU power, lots of ECC ram. If there's going to be an SSD allocated as a cache disk, it's probably expected to be huge and enterprisey too.....
ZFS is awesome, but some features will be disappointing unless you are dealing with adequate resources.