BuildXYZ: Automatic on-demand dependency dispenser – Ryan Lahfa | PackagingCon 2023

Have you ever pondered why our software projects have README to explain how to install them?
That's because it can be hard to automate the installation of the dependencies of a project.

In this work, we will challenge and explore the actual difficulty behind why do we still need READMEs and human instructions to install native dependencies for projects, via a research project, called BuildXYZ, which provides an automatic on-demand dependency dispenser based on a FUSE filesystem that will lazily provide the dependency whenever you actually ask for it on the filesystem.

We will show how such a system performs on tasks such as pip install numpy and relate this to the increasing coupling between application-specific package managers and cross-language dependencies, such as Python library with Rust code, e.g. cryptography.

BuildXYZ is a system with at its heart a FUSE filesystem and a runner that wraps your build system invocation and injects environment variables to influence your program to look into the FUSE filesystem whenever all the previous search paths failed.

For example, consider PATH=/usr/bin:/some/location, if you run a program absent of /usr/bin, it will fallback to look into /some/location, by exploiting this fundamental property of search paths, we can redirect most of the "negative lookups" into a special filesystem where the implementation will trigger search against a package database.

In our case, we decide to use Nix and Nixpkgs as they provide a large software collection and provide on-demand any dependency requested by analyzing the path pattern and checking against a "local database" of resolutions, then, we can even ask the user to choose between multiple candidates, e.g. when analyzing a C++ Boost library path, we can present all the Boost versions in nixpkgs at that time for the user.

BuildXYZ is an open source project hosted at https://github.com/RaitoBezarius/buil... that has multiple aims:

- provide a research platform for the "native dependencies" problem in packaging, e.g. what is the "native dependencies" missing metadata in the application-specific package manager? Can we regenerate the missing information?

- provide a useful tool for developers, packagers to automate further the packaging of existing programs, e.g. https://github.com/nix-community/nix-... is a tool to automatically write Nix packaging expressions, but it relies on heuristics to find about dependencies, combining it with BuildXYZ enable you to learn about much more dependencies.

In the end, we will show multiple usecases that are enabled by BuildXYZ in combination with other tools such as "Lazy Package Management", i.e. only get a program and its dependencies when you actually need it and talk about the performance overhead of such a tool, the previous attempts and what is the outlook for the problem BuildXYZ is trying to solve.
________________________________________________

Student at ENS Ulm in computer science, interested in formal verification and programming languages, NixOS developer and contributor in UEFI (Secure Boot) and boot ecosystems, NixOS release manager for 23.05 and 23.11.

Slightly addicted to Nixpkgs packaging and fascinated by the treasure induced by all of this data.