LCFS: A New Container Filesystem for Modern Datacenters

Re:LCFS: A New Container Filesystem for Modern Datacenters by softwarelimits on Thursday February 02, 2017 @05:08PM (#13555260)

Does anybody know of a good deduplication script / tool that runs on every filesystem and can be used as a nightly cronjob?

[ Parent | on HN ]

Re:LCFS: A New Container Filesystem for Modern Datacenters by kruhft on Thursday February 02, 2017 @06:29PM (#13556044)

Here's a quick one if your filesystem supports hard links:

    #!/bin/ksh

    typeset -A hashes

    find . -type f  -exec shasum -a 512 {} \; |
        while read hash filename; do
            if [[ -z ${hashes[$hash]} ]]; then
                hashes[$hash]="$filename"
            else
                echo ln -f "'${hashes[$hash]}'"  "'$filename'"
            fi
        done

Pipe to /bin/sh if you're happy with the results.

[ Parent | on HN ]

Re:LCFS: A New Container Filesystem for Modern Datacenters by rsanders on Thursday February 02, 2017 @07:28PM (#13556487)

Good answer, but unless you mark the files read-only/immutable, this ought to have big red flags all over it. Userland just can't do this safely in the general case.

[ Parent | on HN ]

Re:LCFS: A New Container Filesystem for Modern Datacenters by kruhft on Saturday February 04, 2017 @04:43PM (#13569260)

Good point. This is for data that's not in use.

[ Parent | on HN ]

Re:LCFS: A New Container Filesystem for Modern Datacenters by khc on Thursday February 02, 2017 @11:18PM (#13557641)

Do you actually use this? Seems at the minimum you want to use -xdev with find and do something better than ln -f.

[ Parent | on HN ]

Re:LCFS: A New Container Filesystem for Modern Datacenters by kruhft on Saturday February 04, 2017 @04:44PM (#13569262)

Yes, I forgot to put in -xdev, but what's wrong with ln -f?

[ Parent | on HN ]

Re:LCFS: A New Container Filesystem for Modern Datacenters by nabla9 on Friday February 03, 2017 @12:57AM (#13558039)

fdupes -rL /dir