Parallelization in The FreeBSD Ports Collection and pkgng utility

by Alexander Pronin for FreeBSD

The FreeBSD Ports Collection has been a primary system for building and installing software on FreeBSD since FreeBSD 1.0. Nevertheless, it does not provide a safe way for building several ports simultaneously. Port’s dependencies are built sequentially. To maintain the system in consistent state while building several ports it is necessary to prevent concurrent access to shared files and directories from multiple processes. My approach is based on lock files, that serve both as barriers and critical section triggers for several concurrent processes. An important aspect of my project is dealing with various failures and unexpected terminations of port’s build process to avoid any deadlock situations and inconsistent state of the ports system. Firstly, my changes to the ports framework allow safe way to build and install several ports at the same time. Secondly, I designed a convenient approach for parallel port’s dependencies builds. The main aim of this project is to make system updates faster and easier. My modifications to the ports collection allow multicore servers to use all its potential both by installing several ports and several port’s dependencies simultaneously. Same goes for the Tinderbox and pointyhat systems used by port committers. Another benefit of my project is package building with pkgng, since the build systems can build packages in parallel.