Building a Debian rootfs from an unprivileged user with debootstrap

At Gatewatcher¹, we put efforts in making our building system reproducible and working offline, so that we can reduce the risk of supply chain attacks. Some efforts are also made so that our building system run with as few privileges as possible.

One of the few things we were still running as a privileged user recently was the build of our initial Debian root filesystem, for our base system and for our containers.

Indeed, the official Debian Docker container from Docker Hub was not generated in a way that we can consider secure for our need. It basically downloads a blob from a web server, does no verification whatsoever of that blob and ships it as the root filesystem². Even though the root filesystem they are using can be rebuilt in a reproducible way, downloading the result from Internet without verifying it against the expected hash is sort of missing the point of reproducible builds. Also, debuerreotype uses debootstrap, which is problematic in itself, as explained hereafter.

To create such root filesystem, multiple tools are provided by the Debian team, among which debootstrap, and multistrap.

Multistrap has not been updated in many years³, and suffers from some limitations that were show-stoppers for us, but it is capable to create root filesystems from an unprivileged user without hacks.

On the other hand, deboostrap is not really friendly with the idea of building a system from an unprivileged user.

First of, there is a check to ensure we are running it with UID 0⁴. This can be bypassed in several documented ways, including using fakeroot, which overloads some libc calls, using LD_PRELOAD. An other, less hacky, way is to run the program in a user namespace.

Unfortunately, this is not sufficient to run debootstrap, since it performs another check consisting of trying to create a “/dev/null” node⁵. This is more problematic since nodes cannot be created from a user namespace, as this would create a easy way of escaping the namespace.

As it seems, though, there is a way to build an unprivileged Debian root filesystem that is even built into deboostrap, using the installation variant “fakechroot”. Alternate code paths exist in deboostrap when this variant is selected that side-step some checks, and fake some calls. This variant also adds a check to ensure this variant is run only if the fakechroot utility is in use. Therefore, you are expected to run debootstap as followed, as documented in the fakechroot manpage:

# apt update && apt install -y debootstrap fakeroot fakechroot
$ fakechroot fakeroot debootstrap --variant=fakechroot bullseye $HOME/rootfs

fakechroot works by overloading some functions with LD_PRELOAD, and has some documented limitations regarding symlinks. As it happens, these limitations include rewrites of absolute symlinks, by prefixing them with the path of the faked chroot. As a result, within the chroot, you will find links that are broken when actually chrooting, such as when you would use that directory hierarchy as a root filesystem on a container or a virtual machine.

With fakechroot:

$ readlink /path/to/my/chroot/usr/sbin/telinit
/path/to/my/chroot/bin/systemctl

Without fakechroot (this is what you want to see, in a normal system):

$ readlink /path/to/my/chroot/usr/sbin/telinit
/bin/systemctl

After some verifications, we decided that it was safe to fake the use of fakechroot, while using the “fakechroot” installation variant. For this, we set the environment variable FAKECHROOT to true, which fakechroot is supposed to set and which is controlled by debootstrap to authorize the use of the “fakechroot” variant. And it worked.

So to build a working root filesystem from an unprivileged user, we are now doing the following:

$ podman unshare
# FAKECHROOT=true debootstrap --variant=fakechroot bullseye chroot/
# tar -C chroot/ --exclude=dev/* -czf ./chroot.tgz .
# exit
$ cat <<EOF > Containerfile
FROM scratch
ADD chroot.tgz .
CMD ["/bin/bash"]
EOF
$ podman build -f Containerfile

This series of commands builds a Debian container from an unprivileged user. More work needs to be done to achieve offline reproducible builds, of course, but none require hacks like this, thankfully.