Continuous Integration

For automatic testing we use GitLab CI. GitLab CI allows us to run runtime tests on GPUs and CPU architectures other than x86, like ARM or IBM POWER. The GitHub repository is mirrored on https://gitlab.com/hzdr/crp/vikunja . Every commit or pull request is automatically mirrored to GitLab and triggers the CI. The configuration of the GitLab CI is stored in the file .gitlab-ci.yml.

Most of the jobs are generated automatically. For more information, see the section The Job Generator.

It is also possible to define custom jobs, see Custom jobs.

GitLab CI uses containers which are already prepared for testing. The containers are built in an extra repository and contain most of the dependencies for vikunja. see the section The Container Registry for more information

All CI related scripts are located at ci/.

Relationship between GitHub.com, GitLab.com and HZDR gitlab-ci runners

Relationship between GitHub.com, GitLab.com and HZDR gitlab-ci runners

The Container Registry

Vikunja uses containers in which as many dependencies as possible are already installed to save job execution time. The available containers can be found here. Each container provides a tool called agc-manager to check if a software is installed. The documentation for agc-manager can be found here. A common way to check if a software is already installed is to use an if else statement. If a software is not installed yet, you can install it every time at job runtime.

if agc-manager -e boost@${VIKUNJA_CI_BOOST_VER} ; then
  export VIKUNJA_CI_BOOST_ROOT=$(agc-manager -b boost@${VIKUNJA_CI_BOOST_VER})
else
  # install boost
fi

This statement installs a specific boost version until the boost version is pre-installed in the container. To install a specific software permanently in the container, please open an issue in the alpaka-group-container repository.

The Job Generator

Vikunja supports a large number of different compilers with different versions and build configurations. To manage this large set of possible test cases, we use a job generator that generates the CI jobs for the different compiler and build configuration combinations. The jobs do not cover all possible combinations, as it would be too much to run the entire CI pipeline in a reasonable amount of time. Instead, the job generator uses pairwise testing.

The stages of the job generator are:

  1. Assembles all input parameters depending on the version.py.

  2. Create the sparse combination matrix. Different rules are applied during the process to avoid invalid combinations, e.g. compiling the CUDA backend with the HIP compiler.

  3. Shuffle the jobs. By default, the pairwise generator has a systematic way of generating the jobs, which is why two consecutive jobs are only slightly different in the results list. Shuffling increases the variety among the first jobs and the chance to find an error early.

  4. Manually reorder jobs. Manually place some jobs at the beginning to increase the chance of finding an error early.

  5. Generate the yaml code for each combination.

  6. Add custom jobs. The yaml code is loaded from files.

  7. Distribute the jobs in waves. Jobs are started in waves because we have a limited number of resources. The waves allow a fair distribution of resources between all projects using the HZDR CI Runner.

  8. Write jobs to a yaml file.

The job generator is located at ci/job_generator/. The code is split into two parts. One part is vikunja-specific and stored in this repository. The other part is valid for all alpaka-based projects and stored in the alpaka-job-coverage library.

Run Job Generator Offline

First you need to install the dependencies. It is highly recommended to use a virtual environment. You can create one for example with the venv-Python module or with miniconda. Once you have created a virtual environment, you should activate it and install the Python packages via:

pip install -r ci/job_generator/requirements.txt

After installing the Python package, you can simply run the job generator via:

# 3.0 is the version of the docker container image
# run `python ci/job_generator/job_generator.py --help` to see more options
python ci/job_generator/job_generator.py 3.0

The generator creates a jobs.yaml in the current directory with all job combinations.

Filter and Reorder Jobs

The job generator provides the ability to filter and reorder the generated job matrix using Python regex. The regex is applied via the commit message for the current commit:

Add function to filter and reorder CI jobs

This commit message demonstrates how it works. The job filter removes
all jobs whose names do not begin with NVCC or GCC. Then the jobs are
reordered. First all GCC11 are executed, then all GCC8 and then the
rest.

CI_FILTER: ^NVCC|^GCC
CI_REORDER: ^GCC11 ^GCC8

The job generator looks for a line starting with the prefix CI_FILTER to filter the jobs or CI_REORDER to reorder the jobs. The filter statement is a single regex. The reorder statement can consist of multiple regex separated by a whitespace. For reordering, the jobs have the same order as the regex. This means that all orders matching the first regex are executed first, then the orders matching the second regex and so on. At the end, all orders that do not match any regex are executed. Attention: the order is only guaranteed across waves. Within a wave, it is not guaranteed which job will start first.

It is not necessary that both prefixes are used. One of them or none is also possible.

Hint

You can test your regex offline before creating and pushing a commit. The job_generator.py provides the --filter and --reorder flags that do the same thing as the lines starting with CI_FILTER and CI_REORDER in the commit message.

Develop new Feature for the alpaka-job-coverage Library

Sometimes one needs to implement a new function or fix a bug in the alpaka-job-coverage library while they are implementing a new function or fixing a bug in the vikunja job generator. Affected filter rules can be recognized by the fact that they only use parameters defined in this globals.py.

The following steps explain how to set up a development environment for the alpaka-job-coverage library and test your changes with the vikunja job generator.

We strongly recommend using a Python virtual environment.

# if not already done, clone repositories
git clone https://github.com/alpaka-group/alpaka-job-matrix-library.git
git clone https://github.com/alpaka-group/vikunja.git

cd alpaka-job-matrix-library
# link the files from the alpaka-job-matrix-library project folder into the site-packages folder of your environment
# make the package available in the Python interpreter via `import alpaka_job_coverage`
# if you change a src file in the folder, the changes are immediately available (if you use a Python interpreter instance, you have to restart it)
python setup.py develop
cd ..
cd vikunja
pip install -r ci/job_generator/requirements.txt

Now you can simply run the vikunja job generator. If you change the source code in the project folder alpaka-job-matrix-library, it will be immediately available for the next generator run.

Custom jobs

You can create custom jobs that are defined as a yaml file. The job yaml files are normally stored in ci/custom_jobs/ and included in the add_custom_jobs() function in ci/custom_jobs/custom_job.py. The custom jobs are added to the same job list as the generated jobs and distributed to the waves.