Re:Syncer: fast stateful file/disk data syncer written on Go
by perlpimp on Sunday September 20, 2015 @03:18PM
(#10248652)
this is great. I wonder how it would perform when I need to sync 10mil. files. Rsync and anything else breaks down at this point, well it does work but it takes forever and rsync bloats to 10-20GB memory foot print and after a few hours starts streaming updates.
[ Parent
| on HN ]
Re:Syncer: fast stateful file/disk data syncer written on Go
by stargrave on Sunday September 20, 2015 @03:37PM
(#10248715)
syncer works on block level, with raw byte sequences. It knows nothing about file systems. Everything is limited with sequential read/write speeds of your hard drives.
It takes blocksize-size memory block for each CPU in your system. I have got 4 CPUs and work with 2 MiB blocks: program will take 8 MiB of RAM.