mboisso

Clients fetch content from stratum 1 servers as needed. Multiple layers of cache are possible/recommended, including 1) a local SQUID proxy (in the case of large infrastructure) 2) the client's local disk 3) a networked/parallel filesystem

Cache invalidation happens through checksums and catalogs (each revision has one or multiple nested catalogs, content is checksummed, and clients can know if one or multiple catalogs are outdated by comparing checksums they have to upstream checksums).

Compute Canada (now the Digital Research Alliance of Canada, http://alliancecan.ca/) has been using CVMFS in production for nearly 7 years now. We use it to distribute all of the software environment used by our researchers on all of our national clusters, as well as to make it available for researchers themselves and university owned computers, and virtual machines or virtual clusters in the cloud. Documentation on using it on any Linux computer here: https://docs.alliancecan.ca/wiki/Accessing_CVMFS.

CVMFS has been essential in making a uniform user-facing software environment available throughout our distributed infrastructure (we have 5 different clusters located across the country, totalizing over 250k cores, close to 2k GPUs, and something like 200PB of disk storage).

It is a proven geo-distributed redundant and high performance (through local caching) filesystem which enables a install-once-available-everywhere strategy. Our software stack currently hosts over 1150 different scientific software packages in over 2800 versions, and built optimized for multiple CPU architectures, totalizing nearly 10k different builds. This is in addition to nearly 10k python packages which we make available in the form of python wheels.

Our software stack now is close to 8TB in size, and contains tens of millions of files.