The first step after having set up, a public CernVM-FileSystem is to install software.

The software on cannot depend on anything that is not on /cvmfs/, this ensures that the software does not lack any runtime dependency and that all the binaries can always run. We call this process, bootstrapping.

Bootstrapping usually involves creating the software in a completely isolated environment in order to avoid introducing hidden dependencies.

However, bootstrapping a software stack is not simple, who compile the compiler? How do you even download the source code for the compiler?

One simple solution would be to ignore this issue, sometimes it is the more pragmatic approach.

However, we opted for a more structured approach.

We build our basic software stack using Nix, a software package manager to build software in a reproducible way.

Nix took care for us of bootstrapping the compilers, but also glib and other fundamental utilities like tar or the coreutils.

However, Nix comes with its set of compromises. All the software build or installed by Nix is installed in /nix/store that does not work well with CVMFS, since it expect software installed in /cvmfs/ .

It is possible to change where Nix install software but compiling yourself Nix. Indeed we recompile it and set the storage directory to /cvmfs/

After having a Nix installation that could write in the directory that we needed, we had to decide how to build the software itself.

The first attempt was to write the software directly on CVMFS.

CVMFS is a read-only filesystem, and for installing software it deploys an overlay mountpoint. The overlay makes possible writing into CVMFS. However, Nix also runs automated tests against the software it builds, and some tests were failing because they were run on top of an overlay filesystem. We could just disable the tests, but we were not so keen to disable tests that should just run fine.

Eventually we opted to create a docker image with our custom Nix installation. Inside the docker image Nix was running, compiling and testing just fine.

Building software on Nix was a generally pleasant and eventless experience. However, some packages were just broken, mostly with the assumption that the final storage of the artifact was /nix/store. The Nix community was somehow helpful, but not the most friendly.

Once all the software was build and installed, we simply compressed it in a tar file and unpack the tar on the CVMFS repository installing the software.

The final step was to copy the forest of symbolic link generated by Nix. For each new package that Nix install a symbolic link in .profile/bin is created. The symbolic link already point in the Nix store, usually /nix/store but in our case /cvmfs/ However, the link itself was in $HOME/.nix-profile/bin.

In order to move also all those links we just exploit the --transform option of tar.

tar --transform='[email protected]/redbeardlab/.nix-profile/@cvmfs/' \

In this transformation we change the paths inside the tarball from home/redbeardlab/.nix-profile to cvmfs/ the links keep correctly pointing to the store in /cvmfs.

After all the software was installed in/cvmfs the bootstrap was complete.

The last step is just to make sure that any compilation will only use the software and the libraries from /cvmfs/

This is done by simply creating a set of symlinks like

/usr/bin/ -> /cvmfs/


/usr/lib -> /cvmfs/

And so on and so forth.


We publish new content each week, subscribe to don't miss any article.