The ATIX development team spent a lot of time in the last Scrum improving the build environment of orcharhino. So far we've been using Jenkins build jobs, which did the job already pretty well. However this approach is not always as flexible as we need it for maintaining a valuable product delivery for our customers. The Jenkins build steps were mainly shell scripts which were directly configured in the Jenkins UI.

Docker

The build process itself utilised Docker-containers. Since the whole structure grew historically, we had a lot of duplicates for nearly identical tasks, generating nearly identical Dockerfiles. This was not a big issue in general, unless we had to make amendements. Depending on the necessary changes this could affect a bunch of files at once, causing a whole series of necessary changes and waste of time and effort. We needed to change and improve this!

After playing around with various sandbox environments based on Jenkins pipelines, we came to the conclusion that moving towards the Jenkins pipeline way, i.e. a "code-based" build environment would be the most elegant solution. As we knew that the upstream Foreman team is successfully using tito / mock to build Foreman, we first took this tool into consideration. However, our conclusion was that building the RPM packages within Docker is at least as good as the tito / mock tool chain.

On top it has the advantage of creating the building environment only once from scratch and reusing it, rather than recreating it for every package which is in the build process. Overall this means we can decrease the RPM building time, using a Docker based approach.

The challenging parts of creating Foreman packages are as follows:

  • there are many different Foreman packages
  • multiple git repositories: The RPM spec file and some resources are stored in foreman-packaging, however the main source code is in the foreman git repository
  • the Foreman build requires a lot of dependencies

The objectives we wanted to achieve with a Jenkins pipeline approach, utilising Docker were:

  • Speed up the overall building process for one specific orcharhino release
  • Everything as code - even the build instructions
  • Reuse the good code from the old Jenkins build steps and rewrite some bad scripts
  • DRY - Don't repeat yourself. No more duplicate codes, Dockerfiles, etc.
  • KISS - Keep it simple stupid, so everyone can understand and improve the whole process
The Foreman ATIX

Now, let me show you how our new process ultimately works:

We are using CentOS 7 as the host system. In case of using another distribution, you may have to adapt some commands.

In case you don't have Docker installed, please add the following repo:

[docker]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/$releasever/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg

Afterwards, install Docker with the following command:


yum install docker-engine

Start and enable the Docker service:


systemctl start docker
systemctl enable docker

The next step is to pull the Docker container from the Docker Hub, as we need it later as a basis for the RPM build container:

docker pull centos

To use the foreman_docker_build script you will also need to install rpm-build and ruby:


yum install rpm-build ruby git

Now it's time to checkout the foreman_docker_build github repository and take a look at the README:

git clone https://github.com/ATIX-AG/foreman_docker_build.git

Change to the foreman_docker_build and start building your Foreman package


cd foreman_docker_build
./build_foreman_packages.sh -n foreman -b 1.15-stable

That's all. After the build is done, the RPMs are located in the directory "foreman/RPM". How does it work internally?

After cloning the foreman-packaging and the Foreman repository the script collects all necessary build files and moves them to foreman/_pkg_build.

Then the interesting part happens.

The script uses erb (embedded ruby) to dynamically create a Dockerfile and the build_rpm.sh script. Those two files are stored in foreman/_pkg_build. The build_rpm.sh is later used inside the Docker container to build the RPM. The Dockerfile contains the build instructions for the Docker container. Since the script is able to create other Foreman packages from the Foreman-packaging distribution as well, the Dockerfile is dynamically generated, based on variables which are located in foreman/_pkg_build/docker_vars.rb. This latter file is written by the build_foreman_packages.sh script.

The next step is to build the Docker container using the Dockerfile and some files in foreman/_pkg_build.
Finally, the Docker container is started to build the RPM. This might take some time, since now the build dependencies from the foreman.spec file are evaluated and installed within the Docker container. In the last step,
the RPM packages are build and stored in "foreman/RPM".

Pretty easy, isn't it?

Link to source code at github