Cover image : David Siglin

TL;DR - You can test if you want to, you can leave your bugs behind

So, you’re working on an application that wants to get included in the CODE-RADE repository, but for some reason, it’s failing in CI. Let’s take the R language for example. This project has a slew of dependencies, some of them optional, such as java, lapack, openblas, pcre2, readline and xz

It is going to be needed soon for the CODATA/RDA DataScience School, but the project is currently Build Status So, you say “shucks, I can fix that !” and set about preparing your pull request to the repository

Before you do that though, you want to test your changes… but how to do that ? This post will shed some light on how to reproduce the testing environment on your laptop or local setup so that you can send pull requests or commits with greater confidence and speed up the CI/CD pipeline.

CODE-RADE targets

CODE-RADE builds are done in a matrix of environments, which define targets. These are constructed using 3 axes :

  • SITE : a variables describing the site for which the application has been compiled, in some qualitative way. Usually set to generic
  • ARCH : the CPU architecture of the target. Typically x86_64
  • OS : the OS of the target. We test for :
    • Ubuntu-14.04
    • Ubuntu-16.10
    • CEntOS-6
    • CEntOS-7

Every application is built for a combination of the size of these dimensions - typically . If different toolchains are used, these add to the dimensions - e.g. different compiler versions, MPI implementations, or pythons - the size of the matrix can grow quite large1. In fact, one of the main benefits of using Jenkins in CODE-RADE is to be able to automate the tests across a wide variety of targets, allowing the developer to focus on writing code instead of manually running checks.

Sometimes, however, you still need to run a few things manually locally, in order to check whether a proposed fix will actually do the trick. Perhaps you don’t have to run the checks for all targets, but just one or two which are representative, or just those nodes which are failing. In order to do this, however, you need to have access to the same environment as is provided on the CI service.

CODE-RADE containers

Luckily, it is now quite easy to deliver execution environments, by building Docker containers. The CODE-RADE containers are built by AnsibleContainer for the specific targets2 and pushed to quay.io3 :

OS Container
CentOS 6 Docker Repository on Quay
CentOS 7 Docker Repository on Quay
Ubuntu 14.04 Docker Repository on Quay
Ubuntu 16.10 Docker Repository on Quay

So, you can run these containers on your local setup and reproduce the CI environment !

Accessing Dependencies

Of course, the containers do not contain the build products they only reproduce the environment - so you need to find some way to include the dependencies in the container when you run the tests.

Luckily for you, you can just mount the CVMFS repository from within the container to have access to it. If it is already mounted on your local setup (of course it is ! why on earth wouldn’t it be !?), you can expose the mountpoint as a volume to the container :

docker run -ti -v /cvmfs:/cvmfs /bin/bash

Of course, once you have access to the dependencies, you need to add the modules in order to have them set up in the shell :

root@2a7757ac74d3:~# . /etc/profile.d/
root@2a7757ac74d3:~# export SITE=generic OS=u1404 ARCH=x86_64
root@2a7757ac74d3:~# module use /cvmfs/
root@2a7757ac74d3:~# module use /cvmfs/
root@2a7757ac74d3:~# module avail gcc

-------------------------------------------------------------- /cvmfs/ --------------------------------------------------------------
gcc/4.9.2 gcc/4.9.4 gcc/5.4.0 gcc/6.3.0

Boom, now you have a choice of compilers.

Back to R

As of at least build 21 R is failing. Let’s suppose we want to have a stab at fixing it. First, we get the repository locally :

git clone

The scripts in the repository explain where to get the source code and how to configure the build. In the case of R, we have the following dependencies :

module add  gcc/5.4.0
module add openblas/0.2.15-gcc-5.4.0
module add lapack/3.6.0-gcc-5.4.0
module add jdk/8u66
module add ncurses
module add readline
module add bzip2
module add xz

and the build is configured as :

CFLAGS="${CFLAGS} -I${BZLIB_DIR}/include -I${XZ_DIR}/include" \
LDFLAGS="-L${BZLIB_DIR}/lib -L${XZ_DIR}/lib -llzma" \
../configure \
--build=x86_64-pc-linux-gnu \
--host=x86_64-pc-linux-gnu \
--target=x86_64-pc-linux-gnu \
--prefix=${SOFT_DIR} \
--enable-static \
--enable-shared \
--with-readline=no \
--with-x=no \
--with-blas \
--with-lapack \

Taking a look at the issue or the console output, we can see that the problem is linking against PCRE4 :

checking whether PCRE support suffices... configure: error: pcre >= 8.10 library and headers are required

Seems pretty easy to fix - just include -L${PCRE_DIR}/lib in the LDFLAGS and -I${PCRE_DIR}/include in the CFLAGS.

You can try that first in the container.

Speeding up manual builds

Finally, we would like to be able to run these things with a single command. Currently, a few manual commands need to be done interactively in order to set up the modules, etc. This could be improved by having a script in the repo which when run just passed the local CVMFS mount as a volume to the container, as well as the local working directory where the -deploy repository was cloned. In this way, one could just run without having to do things manually. Similarly, one could pass environment variables and scripts to the docker run command in order to set up the environment properly… We’ll comment in this at a later stage maybe.

For the meantime, creating an environment in which to interactively test the build could be done with :

docker run -ti -v /cvmfs:/cvmfs -v pcre-deploy:/workspace /bin/bash

Debugging just got a whole lot more responsive and more keyboard-slammier :keyboard:


  1. The NumPy job e.g. has cells in the matrix. 

  2. Bruce Becker, “AAROC/CODE-RADE-build-containers: CODE-RADE Foundation Release 3 - Build Containers”. Zenodo, 06-May-2017. DOI 

  3. We chose Quay for the added security scanning and data analytics services, over DockerHub. 

  4. Perl Compatible Regular Expressions 

Previous Entry : It's a hard-link life