Cover image : Image courtesy of New Old Stock - http://nos.twnsnd.co/image/69699793933

TL;DR - Two releases coming up : Where we at ?

A recent post discussed the roadmap for the CODE-RADE project, and put forward two important dates for releases :

Here’s a quick update of where we’re at1

Things in

Abinit

Abinit is configured to build for:

  • OS : u1404, sl6
  • VERSION : 7.10.5
  • GCC Version : 4.9.2, 5.1.0

All of the dependencies of Abinit are passing :

  • fftw3
  • gsl
  • lapack
  • netcdf

This is a good place to be, but abinit itself is failing on all targets due to the FFTW3 dependency. The error is described in #4.

TODO - check what kind of FFT Abinit wants.

GADGET-2

GADGET is configured to be compiled with

  • fftw2
  • openmpi
  • gsl
  • hdf5

The job is not yet complete, since the check-build.sh and deploy.sh scripts are missing. There seems to be some misconfiguration with the makefiles, which need to be checked.

GROMACS

Gromacs is configured for the targets:

  • VERSION : 5.1.1
  • GCC Version : 4.9.2, 5.1.0
  • OpenMPI version : 1.8.1, 1.8.8

Gromacs depends on

  • boost
  • fftw3
  • lapack

as well as the aforementioned GCC and OpenMPI. GROMACS is currently suffering from the wrong FFTW configuration. The build is configured as :

cmake ../ \
-G"Unix Makefiles" \
-DCMAKE_C_COMPILER=mpicc \
-DCMAKE_CXX_COMPILER=mpicxx \
-DGMX_X11=OFF \
-DGMX_FFT_LIBRARY=fftw3 \
-DGMX_DOUBLE=OFF \
-DGMX_GPU=OFF \
-DGMX_OPENMP=ON \
-DGMX_MPI=ON \
-DGMX_EXTERNAL_BLAS=on \
-DGMX_BUILD_MDRUN_ONLY=ON \
-DCMAKE_PREFIX_PATH=${BOOST_DIR}/boost;${LAPACK_DIR};${FFTW_DIR};${OPENMPI_DIR} \
-DREGRESSIONTEST_DOWNLOAD=ON \
-DCMAKE_INSTALL_PREFIX=${SOFT_DIR}/${VERSION}-gcc-${GCC_VERSION}-mpi-${OPENMPI_VERSION}

However, this is failing with the following error message :

– pkg-config could not detect fftw3f, trying generic detection Could not find fftw3f library named libfftw3f, please specify its location in CMAKE_PREFIX_PATH or FFTWF_LIBRARY by hand (e.g. -DFFTWF_LIBRARY=’/path/to/libfftw3f.so’) CMake Error at cmake/gmxManageFFTLibraries.cmake:81 (MESSAGE):

Cannot find FFTW 3 (with correct precision - libfftw3f for mixed-precision GROMACS or libfftw3 for double-precision GROMACS). Either choose the right precision, choose another FFT(W) library (-DGMX_FFT_LIBRARY), enable the advanced option to let GROMACS build FFTW 3 for you (-DGMX_BUILD_OWN_FFTW=ON), or use the really slow GROMACS built-in fftpack library (-DGMX_FFT_LIBRARY=fftpack).

The error doesn’t seem to be in CMAKE_PREFIX_PATH, so it is likely in one of the FFTW variables.

TODO : Find out how to specify the correct FFTW3 configuration

Update : 10/02 - fixed in d8e48ec5, however GROMACS in the repo only contains mdp_mpi and none of the other executables. Need to revisit the configuration.

Python

Python is one of the generic frameworks which is difficult to treat like the other applications in CODE-RADE, because it has it’s own package management tools and internal dependency management. It also doesn’t usually deal with modules, rather using it’s own virtualenv to deal with different versions of Python and different modules within it. However, it can still be built from source, and the supporting applications which it uses - pip, easy_install, etc can be added to it. Care should be taken to use the python environment variables properly in order to ensure that the subsequent python modules such as those in the SciPy suite, are installed and delivered correctly.

The current overview of the status of python is :

Snapshot of the status of Python at build 26 - current at the time of writing

Python currently depends on

  • bzlib
  • ncurses
  • readline
  • sqlite
  • tcltk
  • zlib-deploy

which are all currently passing.

Python 2.6 and Python 2.7 are failing as follows.

Python 2.6

Python 2.6 needs setuptools. This is downloaded as part of the build script, but fails immediately with :

Traceback (most recent call last):
File "setup.py", line 21, in <module>
exec(init_file.read(), command_ns)
File "<string>", line 11, in <module>
File "/var/lib/jenkins/workspace/python-deploy/ARCH/x86_64/GCC_VERSION/4.9.2/NAME/python/OS/u1404/SITE/generic/VERSION/2.6.9/Python-2.6.9/setuptools-18.3.2/setuptools/__init__.py", line 12, in <module>
from setuptools.extension import Extension
File "/var/lib/jenkins/workspace/python-deploy/ARCH/x86_64/GCC_VERSION/4.9.2/NAME/python/OS/u1404/SITE/generic/VERSION/2.6.9/Python-2.6.9/setuptools-18.3.2/setuptools/extension.py", line 8, in <module>
from .dist import _get_unpatched
File "/var/lib/jenkins/workspace/python-deploy/ARCH/x86_64/GCC_VERSION/4.9.2/NAME/python/OS/u1404/SITE/generic/VERSION/2.6.9/Python-2.6.9/setuptools-18.3.2/setuptools/dist.py", line 16, in <module>
from setuptools.depends import Require
File "/var/lib/jenkins/workspace/python-deploy/ARCH/x86_64/GCC_VERSION/4.9.2/NAME/python/OS/u1404/SITE/generic/VERSION/2.6.9/Python-2.6.9/setuptools-18.3.2/setuptools/depends.py", line 6, in <module>
from setuptools import compat
File "/var/lib/jenkins/workspace/python-deploy/ARCH/x86_64/GCC_VERSION/4.9.2/NAME/python/OS/u1404/SITE/generic/VERSION/2.6.9/Python-2.6.9/setuptools-18.3.2/setuptools/compat.py", line 28, in <module>
import urllib2
File "/apprepo/generic/u1404/x86_64/python/2.6.9-gcc-4.9.2/lib/python2.6/urllib2.py", line 93, in <module>
import hashlib
File "/apprepo/generic/u1404/x86_64/python/2.6.9-gcc-4.9.2/lib/python2.6/hashlib.py", line 138, in <module>
sha224 = __get_builtin_constructor('sha224')
File "/apprepo/generic/u1404/x86_64/python/2.6.9-gcc-4.9.2/lib/python2.6/hashlib.py", line 66, in __get_builtin_constructor
import _sha256
ImportError: No module named _sha256

It seems that Python has not built the sha256 library.

Python 2.7

Python 2.7 seems to have passed build and check-build, but when cleaning the build directory at the start of deploy we get the following error :

+ ./deploy-2.7.11.sh
rm: cannot remove `build/lib.linux-x86_64-2.7': Directory not empty

This is thrown by the lines

cd ${WORKSPACE}/Python-${VERSION}/build-${BUILD_NUMBER}
rm -rf *

This could easily be circumvented by a quick hack, but the origin of the problem is not understoo (yet)

Quantum Espresso

Quantum Espresso is having issues compiling against the LAPACK libraries provided.

WRF

The Weather Research and Forecasting model is very widely used in meteorology and climatology. It depends on

  • hdf5 (compiled with MPI)
  • jasper
  • mpich
  • netcdf (compiled with MPI)

all of which are passing.

The project files are not yet ready, so this build is failing to find the scripts. Although the build has been tested locally using the artifacts in fastrepo, this has only been done by hand and the scripts have not yet been committed to the repo.

The rest

There are several applications which have been proposed for the Public Release. Apart from those described above, we have the list below. A short summary of the status of each is given.

Name Dependencies Comment
GAMESS HDF5 Dependencies all fulfilled, but not tested locally
Repast Symphony Java Dependencies all fulfilled, but not tested locally
ABySS BOOST, Sqlite, SparseHash SparseHash is not yet in Foundation Release, but is WIP.
OpenFOAM BOOST, OpenMPI, gnuplot, readline, ncurses gnuplot is not yet in Foundation Release.. The OpenFOAM job and repo are present, but only the build script is prepared.
AutoDock None, apparently Low hanging fruit ? Need to add the scripts to the repo and create the Jenkins job.
Velvet OpenMP Low hanging fruit ? Need to add the scripts to the repo and create the Jenkins job.
OASES OpenMP, zlib, Velvet Low hanging fruit ? Need to add the scripts to the repo and create the Jenkins job.
WEKA Java JDK JDK has been added with build 113 of fastrepo and WEKA has been tested manually. Compilation is not clear yet.
PLink zlib (uses it's own compiled version), LAPACK Manual tests have passed.
Pythia8 Several CLHEP and CERN libraries necessary This one will take a while still.
FreeFem++ GSL, OpenMPI/MPICH, PETSc, LAPACK, BLAS, SuiteSparse, FFTW3, HDF5. Most of these can be downloaded by the compilation scripts proivded by the package, but that would be a bit couter to the CODE-RADE philosophy.
CosmoMC pmclib, CAMB, GSL, FFTW3, LAPACK, WMAP, fitsio WMAP, pmclib, CAMB and WMAP are missing. The rest have been integrated. CAMB needs HealPix and Fisher. Work has started on Healpix
Gadget 2 FFTW2, GSL, HDF5 Work has started, the repo has been created as well as the job. Check and deploy scripts are still to be committed. FFWT2 needs to be compiled with MPICH, it is currently only compiled with OpenMPI
HTK none Low hanging fruit ? Need to add the scripts to the repo and create the Jenkins job.
MITML none Low hanging fruit ? Need to add the scripts to the repo and create the Jenkins job.
GAMA-Platform Java JDK JDK has been added with build 113 of fastrepo and Gama has been tested manually. Compilation is not clear yet.
IMA-2 none Low hanging fruit ? Need to add the scripts to the repo and create the Jenkins job.

Next steps.

The applications which have been described as low-hanging fruit are going to be added over the next few days. Significant work needs to be done for a few of the other ones, and lower-lying dependencies need to be added and compiled for different targets. We look forward to having some contribution from those interested, and using some of these as example applications in the upcoming Sci-GaIA winter school.

For now, let’s keep

Footnotes and References

  1. We’ll report only on the user applications in the Public Release