Creating a CI/CD pipeline with GitHub and Multibranch Pipelines in Jenkins

My intention here is to show how to set up a simple CI/CD multibranch pipeline for teams looking to explore the continuous integration and continuous delivery end of DevOps. This pipeline provides a starting point which can be changed and expanded depending on particular team requirements.

This article assumes your team is already familiar with git and GitHub, and that you have Jenkins installed and ready to use in a location accessible from GitHub. Installing Jenkins is quite straightforward so I won’t go into that here, as there are plenty of other guides available for that. I also won’t spend time explaining what CI/CD is, because again there’s plenty of info about that out there, and if you’re looking for implementation guides then you probably already know what CI/CD is anyway and just want to get started.

This pipeline uses three branches in the git repository: dev, test and main. The dev branch is for development builds. Upon creation of a pull request and successful merge from dev into the test branch, the test branch will be used to run a simple automated test. Again, after a successful pull request/merge from test to main, the main branch is used for delivery to a staging environment for QA testing. This is quite basic and can be changed and expanded according to team needs, e.g. feature branches for specific areas of code, additional test environments, the addition of deployment to a production environment, etc.

The pull requests and merges are done manually so that code can be reviewed and checked for issues before merging. Apart from that, the rest of the builds, tests, and deliveries/deployments are automated.

GitHub repository creation

For the purposes of this task, it’s probably best to create a new repository to keep things clean and simple. So, create a new repository in GitHub now. You’ll most likely want to make this a private repository, unless you need it to be public for some specific reason. Choose the option to Add a README file as that will remove a few complications further down the line. Once it’s created, don’t do anything else with it yet.

For the remainder of this article I’ll assume this repository is called multibranch-pipeline-test. I’d advise using this name to keep things simple, but you can call it something else if you need to – just change the name accordingly in the examples below containing the repository name.

GitHub Webhook setup

Your GitHub repository needs to know how to send notifications to Jenkins to trigger pipelines. This is configured by going to Settings -> Webhooks in the repository on GitHub, then clicking the Add webhook button, at which point you should see something like this:

Payload URL consists of your Jenkins URL followed by github-webhook/. For example, if your Jenkins URL is https://mydomain.com/jenkins/ then the Payload URL will be https://mydomain.com/jenkins/github-webhook/. Content type is application/json. Keep SSL verification enabled unless there’s a very good reason not to (disable at your own risk). For events, I think it may only be necessary for this particular task to send just the push event, but I like to keep options open for future possibilities so I choose Send me everything. Click the Add webhook button to finish the setup.

Pipeline discovery setup

A new SSH keypair is needed so Jenkins can talk to GitHub to discover branches and scan content therein. Run the following command locally to create a keypair to use:

ssh-keygen -f jenkins-github

Use an empty passphrase (i.e. just hit Enter when prompted for a passphrase). This will create the files jenkins-github containing the private key, and jenkins-github.pub containing the public key. (You’ll probably want to move these files to a safe and secure folder so you have them around for future reference. Also, do not keep these files in your GitHub repository as that should not be considered secure.) In your GitHub repository, go to Settings -> Deploy keys, then click the Add deploy key button. You should see something like the following:

Give it the title jenkins-github to make it easier to keep track of keys. Then, copy the contents of the public key file jenkins-github.pub you created earlier and paste it into the Key box. You can see that GitHub is giving you a guideline to help ensure you are pasting the correct text in here. Once done, click the Add key button.

Repository branch creation

New branches can be created within GitHub if you prefer, but here I’ll do it in the local development environment, on the command line.

Start by cloning your new repository using the details shown when you click the Code dropdown button on your GitHub repo (yourgithubaccount should be your GitHub account name):

git clone git@github.com:yourgithubaccount/multibranch-pipeline-test.git

cd to the multibranch-pipeline-test folder, create the new branches (test and dev), and push the new branches to the GitHub repository:

cd multibranch-pipeline-test
git checkout -b test
git push -u origin test
git checkout -b dev
git push -u origin dev

Creation of Multibranch Pipeline in Jenkins

In Jenkins, go to New Item, then you should see something like the following:

To keep things consistent, it’s best to give this job the same name as your GitHub repository (multibranch-pipeline-test if you’re using the same naming as I am). Choose Multibranch Pipeline and click the OK button. You should then encounter a Configuration page, and the first thing we’re interested in here is the Branch Sources section, under which you need to click the Add source button and choose Git (not GitHub), at which point you should see something like this:

For Project Repository, use the details shown when you click the Code dropdown button on your GitHub repo (i.e. git@github.com:yourgithubaccount/multibranch-pipeline-test.git, where yourgithubaccount should be your GitHub account name). Then it’s necessary to add credentials to allow Jenkins to talk to your GitHub repository, so click the Add button, click on multibranch-pipeline-test (this is the Folder Credentials Provider for this items in Jenkins), then this page should pop up where you can add credential details:

Here we will add the private key from the SSH keypair we created earlier. So, under Kind choose SSH Username with private key. For the ID, keep the keypair name consistent across locations, so put jenkins-github. The Username should be git (as that’s the SSH username GitHub uses by default). Then, under Private Key, choose Enter directly, then you will have the option to Add, then paste in the contents of the private key file jenkins-github (not jenkins-github.pub) created earlier. Once that’s done, click the Add button and you should be taken back to the Configuration page. Once there, ensure git is selected for Credentials so you’re using the private key you just added.

The only other on this Configuration page I’d suggest changing for now is Scan Multibranch Pipeline Triggers, so that discovery scanning happens automatically:

I set this to 5 minutes which is probably fine for now. It can always be changed later according to needs/preferences. Once that’s done, click the Save button on this page. It should immediately do a discovery scan of the GitHub repository, and the log output should appear. You should see some output like this at the bottom of the log:

A strong clue is visible here for what needs to be done next: create a Jenkinsfile.

Creation of Jenkinsfile for Multibranch Pipeline stages

The Jenkinsfile defines the pipeline configuration, so that Jenkins knows what to do for each branch it finds. When scanning a branch, Jenkins looks for a Jenkinsfile and, upon finding one, follows the instructions for that branch contained therein.

In the local copy of the repository, check you are in the dev branch:

git branch

You should see:

* dev
  main
  test

If dev is not selected (shown by the asterisk to the left) then change to the dev branch:

git checkout dev

Next, in your repository create a file called Jenkinsfile with the following contents:

pipeline {
  agent any
  triggers {
    githubPush()
  }
  stages {
    stage('Tools') {
      steps {
        sh 'pandoc -v'
      }
    }
    stage('Build') {
      steps {
        sh '_build/app.sh'
      }
    }
    stage('Test') {
      when {
        branch 'test';
      }
      steps {
        sh '_test/app.sh'
      }
    }
    stage('Deliver') {
      when {
        branch 'main';
      }
      steps {
        sh '_deliver/app.sh'
      }
    }
  }
}

This firstly defines a general trigger “githubPush” which uses the GitHub webhook we set up earlier to send a notification to Jenkins, which then takes action when a push (or potentially other type of event) occurs. Then there are four stages:

1. Tools – checks dependencies to ensure requisite tools are installed for a successful build (in this case, pandoc).
2. Build – performs the build for whichever branch currently being acted on.
3. Test – runs the test(s) when acting on the test branch.
4. Deliver – delivers the app for QA testing when acting on the main branch.

Setup of pipeline stages (Tools and Build) for the dev branch

Having added the Jenkinsfile in the dev branch, let’s set things up so that Jenkins can use this to perform a successful build on this branch. There shouldn’t be an awful lot to do for the Tools stage as there is only one dependency: pandoc. If pandoc is not found, the pipeline will fail at this first stage, so it would be best to install it on the Jenkins server now (apt install pandoc on Ubuntu; yum install pandoc on Red Hat/CentOS if you have the EPEL repository set up).

For the purposes of demonstrating the Build stage and the other stages afterwards, we’re going to build an extremely basic “app” in our repository which will consist simply of some HTML which can be served via HTTP by a web server. Make sure you’re still on the dev branch, as described above.

Create a Markdown (text) file called app.md containing the following:

# My App

This will do some cool stuff.

Next we will create the automated build script, which in this case will convert the Markdown to an HTML file. Create a folder called _build, then create the shell script file _build/app.sh containing the following:

#!/bin/bash

echo "Building HTML from Markdown"
pandoc -o app.html app.md

Make the script executable:

chmod 755 _build/app.sh 

Then you can try running it locally:

_build/app.sh

If you don’t have pandoc installed you’ll get an error, in which case install it (brew install pandoc on macOS; apt install pandoc on Ubuntu) and try again. When successful, you should see:

Building HTML from Markdown

The file app.html should now be present, containing the following:

<h1 id="my-app">My App</h1>
<p>This will do some cool stuff.</p>

We now have our very basic app with build script, which we can use as the basis for our pipeline. We don’t really want the HTML file kept in the repository though, so create a .gitignore file and add an exclusion for HTML files:

*.html

Add all new content to the dev branch of the repository, and commit the changes:

git add -A
git commit -a -m "Adding Tools and Build"

Pushing to GitHub to trigger dev build

On the Status page for the multibranch-pipeline-test job in Jenkins, the following should be visible currently:

In the local repository, push the changes to GitHub:

git push

Then in Jenkins, watch the Build Queue and Build Executor indicators in the bottom left. Shortly a scan should occur, followed by a build. Refresh the Status page and this should now be visible:

The dev branch has now appeared because Jenkins has done a successful scan and found a Jenkinsfile in that branch. In turn, the directives in the Jenkinsfile have triggered the Tools scan and the Build script (_build/app.sh). Click on dev and this should be visible:

We can see what the pipeline has done, including details of the successful Tools check and Build. More details can be viewed, including logs. For example, hover the mouse over the green rectangle in the Build column, and a Logs option will appear. Click to view the logs and this should appear at the top:

We see which script was run, and the build output.

We see no activity yet on the Test and Deliver phases, because in the Jenkinsfile those steps are not run for the dev branch.

Setup of test branch with an automated test

In the local repository, we’ll stay in the dev branch for now and add our simple test. Create a folder called _test and add the file _test/app.sh containing the following:

#!/bin/bash

echo "Starting HTTP server"
nohup python3 -m http.server &
pid=$!
sleep 1

echo "Checking for string in HTTP response"
curl -s -S "http://127.0.0.1:8000/app.html" | egrep "<h1 id=\"my-app\">My App</h1>" > /dev/null
result=$?

echo "Killing HTTP server"
kill $pid
[ $result == 0 ] || exit 1

Then make sure it’s executable:

chmod 755 _test/app.sh

This test script uses Python 3 to fire up a temporary web server, checks that the HTML file app.html is served by the web server and that the title appears correctly in the output, then shuts down the temporary web server. If it did not receive the correct output then it will fail with an exit status of 1, which in turn means Jenkins will show it as a failed test.

Run it locally now to try it:

_test/app.sh

The output should be as follows:

Starting HTTP server
nohup: appending output to 'nohup.out'
Checking for string in HTTP response
Killing HTTP server

We don’t want the nohup.out in the repository, so add it to the .gitignore file:

*.html
nohup.out

Add the files to the branch, commit, then push:

git add -A
git commit -a -m "Adding Test"
git push

The push will trigger a build in Jenkins (triggered by the GitHub webhook we set up earlier), but it won’t look any different to last time because the test isn’t run for the dev branch.

Merging via pull request, and triggering Test stage in pipeline

In a team, the changes made on the dev branch might need to be reviewed before passing them for testing in Jenkins. The developer who made the changes would therefore submit a pull request.

In the repository on GitHub there should be a message like this:

Click the Compare & pull request button, then this screen should appear:

Make sure base is set to the test branch (not main) and compare is set to the dev branch, then click Create pull request. A screen like this should appear:

We’ll assume here the changes have been reviewed satisfactorily, so hit the Merge pull request button and confirm the merge. Then watch Jenkins and you should see it scan then build, at which point if you refresh the Status page for the multibranch-pipeline-test job, it should now look like this:

The test branch has now appeared, because the content has been merged from the dev branch so it now contains the Jenkinsfile. Clicking on test should show that, in addition to the Tools and Build stages, it’s also run the Test stage as defined in the Jenkinsfile for the test branch:

As before, the log can be checked and it should show something similar to the output seen when the test script was run in the local repository:

Setting up the Deliver stage in the pipeline, for delivery to QA

Now we have successful Build and Test stages, we can move on to the Deliver phase for delivering our app to a QA environment for testing by humans.

To keep things straightforward for this example, I’m going to assume there’s a QA environment set up on the Jenkins server, with a web server delivering content from the virtual host https://qa.mydomain.com/ using application content located in the folder /var/www/qa.mydomain.com. Hopefully you can set up something similar, and you’ll need to change the domain names and modify other details according to your own setup. Obviously this can be expanded to deployments on other servers and so forth, but this should suffice for this simple example.

In the dev branch in the local repository, create the folder _deliver then add the file _deliver/app.sh, making sure it’s executable. The contents of the file can be something like this, modified as needed for your own setup:

#!/bin/bash

cp -f app.html /var/www/mydomain.com

This simply copies the built HTML file to a place where it can be served by the web server for QA engineers to check the app. Now add, commit and push:

git add -A
git commit -a -m "Adding Deliver"
git push

In GitHub, do a pull request and merge from the dev branch to the test branch, as before.

Then do another pull request, this time setting base to the main branch, and compare to the test branch. Assume that this is all reviewed satisfactorily and merge the changes to the main branch. Fairly soon, Jenkins will do its thing, and – since the main branch now has a Jenkinsfile – the main branch will appear on the Status page:

Clicking on main should hopefully show that a successful build has run and that the Deliver stage has been successful (there’s no Test stage for automated testing here, as this was excluded from the main branch in the Jenkinsfile):

As before, the logs for each stage can be checked for details.

By visiting https://qa.mydomain.com/app.html (modify the domain/path as needed for your setup) in a web browser, we can now see the delivered app, ready for the QA engineers to check:

Conclusion

A Multibranch Pipeline has been created in Jenkins using a GitHub repository with three branches: dev, test and main. This provides the following workflow:

1. Developers can update their code in the dev branch, push their changes to check it’s successfully built in Jenkins in the Tools and Build stages, then raise a pull request for merging to the test branch.
2. After a successful review, the dev branch is merged to the test branch, at which point Jenkins will build, run the test(s) in the Test stage and show the result(s). If all goes well, a pull request can be raised for merging to the main branch.
3. Once the test branch is merged to the main branch, Jenkins will build then run the Deliver stage to deliver the app for QA testing.

As I mentioned earlier, this can be changed and expanded according to team needs, e.g. feature branches for specific areas of code, additional test environments, the addition of deployment to a production environment, etc. There are many other tools and features that can be used in Multibranch Pipelines, which can be added as needed for compiling code, building Docker containers, provisioning temporary AWS environments, etc.

It’s quite basic in its current form, but it hopefully shows the potential power and flexibility of Multibranch Pipelines for effective continuous integration, continuous delivery, and continuous deployment, and it gives a solid starting point for expanding as needed.

If you require assistance with Multibranch Pipelines, other areas discussed in this article, or any other DevOps and infrastructure issues, contact me to explain your needs and we can discuss consultancy and freelance options.