Configuring a CD pipeline for your Jenkins CI

Home / Configuring a CD pipeline for your Jenkins CI

Overview

Jenkins is a very popular Java-based open source continuous integration (CI) server that allows teams to continuously build applications across platforms. Azure Pipeline includes the ability to build any application on any platform including Windows, Linux and Mac. However, it also integrates well with Jenkins for teams who already use or prefer to use Jenkins for CI.

There are two ways to integrate Jenkins with Azure Pipelines:

One way is to run CI jobs in Jenkins separately. This involves the configuration of a CI pipeline in Jenkins and a webhook in Azure DevOps that invokes the CI process when source code is pushed to a repository or a branch.
The alternate way is to wrap a Jenkins CI job inside an Azure pipeline. In this approach, a build definition will be configured in Azure Pipelines to use the Jenkins tasks to invoke a CI job in Jenkins, download and publish the artifacts produced by Jenkins.

An Azure CD pipeline can be configured to pick these build artifacts, irrespective of the approach, for deployment. While there are pros and cons with both the approaches, the latter approach has multiple benefits:

End-to-end traceability from work item to source code to build and release
Triggering of a Continuous Deployment (CD) when the build is completed successfully
Execution of the build as part of the branching strategy

What’s covered in this lab

This lab covers both the approaches and the following tasks will be performed

Provision Jenkins on Azure VM using the Jenkins template available on the Azure Marketplace
Configure Jenkins to work with Maven and Azure DevOps
Create a build job in Jenkins
Configure Azure Pipeline to integrate with Jenkins
Configure a CD pipeline in the Azure Pipelines to deploy the build artifacts

Before you begin

Microsoft Azure Account: You will need a valid and active Azure account for the Azure labs. If you do not have one, you can sign up for a free trial
Putty a free SSH and Telnet client
Set up your Azure DevOps project using the MyShuttle template in the Azure DevOps Demo Generator. We will use a Java web app that connects to a MySQL backend.

Setting up the Jenkins VM

To configure Jenkins, the Jenkins VM image available on the Azure Marketplace will be used. This will install the latest stable Jenkins version on an Ubuntu Linux VM along with the tools and plugins configured to work with the Azure. Click on the Deploy to Azure button below to get started. Note: After clicking create, choose Password as the Authentication Type. Make note of the username and password.
Once the Jenkins VM is provisioned, navigate to the VM’s overview page and make a note of the public IP address. This information will be required to connect to the Jenkins VM from Putty

Note: Jenkins, by default, listens on port 8080 using HTTP. To configure a secure HTTPS connection, an SSL certificate will be required. If HTTPS communication is not being configured, the best way to ensure that the sign-in credentials are not leaked due to a “Man-in-the-middle” attack is by logging-in using the SSH tunnelling. An SSH tunnel is an encrypted tunnel created through an SSH protocol connection, that can be used to transfer unencrypted traffic over an unsecured network.
To initiate an SSH tunnel, the following command needs to be run from a Command Prompt. An SSH tunnel creates a secure connection between your host and remote computer through which services can be relayed. If this command is successful, you should be able to access the remote Jenkins on port 8080 on your local machine. Note: If you have a website on 8080, you may need to choose another port.
```
 putty.exe -ssh -L 8080:localhost:8080 <username>@<ip address>
```
Note: To run the above command, either the Putty.exe needs to be placed in the path selected in the Command Prompt or the full path of the Putty.exe need to be provided in the command.
Log in with the user name and password that you have provided while provisioning the Jenkins VM.
Once the connection is successful, open a browser on the host machine and navigate to the URL http://localhost:8080. The Sign in page for Jenkins will be displayed.
For security reasons, the application credentials are stored in a standalone file in the VM. This credentials will need to be retrieved and provided to sign in to Jenkins. Return to the Putty terminal and type the following command to open the credentials file and copy the user name and password.

sudo cat /home/bitnami/bitnami_credentials
Return to the browser, paste the copied Username and Password to sign in to the Jenkins.

Installing and Configuring Plugins

We will now install the Maven and VSTS (yet to be renamed Azure DevOps!) plugins that we require for this lab. Click Manage Jenkins on the Jenkins home page and select Manage Plugins. Select the Available plugins tab and search for team services
Select VS Team Services Continuous Deployment plugin and select Install without restart
Select Manage Plugins, select the Available tab and search for maven-plugin
Select the Maven Integration Plugin and select the Install without restart button to install the plugin.
Once the plugin is installed, go back to Manage Jenkins and select the Global Tool Configuration option.

Note: Jenkins provides great out-of-the-box support for Maven. Since Maven is not yet installed, it can be manually installed by extracting the tar file located in a shared folder. Alternatively, when the Install automatically option is selected in the Global Tool Configuration screen, Jenkins will download and install Maven from the Apache website when a build job requires it.
To install Maven, go to Maven section and click on Add Maven option and provide a name. Then select the Apply button. The latest version of Maven at the time of writing this lab is 3.8.6
Select the Back to Dashboard button to return to the home page. We are done with the setup. Let’s go and create a new CI job.

Creating a new build job in Jenkins

From the Jenkins home page, click on the New Item link. Provide a name for the build definition, select Maven project and click OK.
Now scroll down to the Source code Management section. Select Git and provide the clone URL of the Azure DevOps Git repo in the format http://dev.azure.com/{your org name}/{team project name}/_git/MyShuttle. If you do not see Git under Source code management (not a usual thing), you will need to install/enable the Git plugin.
Your Azure repo is very likely to be private. Unless you have a public repo, you should provide the credentials to access the repository. If you do not have one or don’t remember the credentials, go to your Azure Repos and select the Clone option. Select Generate Credentials and note down Username and Password.
Select the Add | Jenkins option to add a new credential. Provide the Username and Password created in the previous step and click the Add button to close the wizard
Select the credential created in the previous step from the drop-down. The error message should disappear.
The source code for this application has both unit tests and UI tests. We are not ready to run the UI test at this point. So, we will specify to run only the unit tests. Scroll down to the Build section and provide the text package -Dtest=FaresTest,SimpleTest in the Goals and options field.
Once the build is complete, you can specify what action you want to take. For instance, you can archive the build artifacts, trigger an Azure CD pipeline, deploy directly to Azure App Service, etc., We will choose the Archive the artifacts option in the Post-build Actions.

Note: Note there is also Post-build steps section which is very similar to the actions section. The tasks configured in the post-build steps/actions are executed after all the build steps have been executed.
Enter target/*.war, *.sql in the Files to archive text box. Select the Save button to save the settings and return to the project page.
The configuration is now completed, Select the Build Now option to initiate an Ad-hoc build. The build progress will be displayed on the left pane in the Build History section
To view the build details and the list of build artifacts, select the build number displayed in the Build History section.
Select the Test Result option to view the results of the unit tests that were included in the build definition.

Next, we will explore the two different options available to trigger the Jenkins CI job when a code is pushed to Azure Repos.

Preparing Jenkins machine to use service hook in Azure DevOps

For the service hook in Azure DevOps to work - Jenkins machine should accept incoming connections to port 8080.

Open https://portal.azure.com/ and access your virtual machine with Jenkins.
Click the “Networking” link in the Settings tab and click “Add inbound port rule” button
Set an inbound rule for port 8080
Navigate to your Jenkins page and go to User | Configure. Click on Add new token under API Token section and give some name and click Generate. Make a note of the Token generated. We would be using this Token as Jenkins password in Azure DevOps. Save the settings

Approach 1: Triggering the CI via a service hook in Azure DevOps

In this approach, a service hook will be configured in Azure DevOps to trigger a Jenkins build upon a code commit. Service hooks enable you to perform tasks on other services when events happen in your Azure DevOps Services projects.

To configure the service hook, navigate to the Azure DevOps project settings page and select Service hooks under General. Select + Create subscription.
In the New Service Hook Subscriptions screen, select the Jenkins option and then click the Next button. Jenkins service supports three events - Build completed, Code Pushed and Pull request merged. We are only interested in the code push event - so, select Code pushed for the Trigger on this type of event field. Select the MyShuttle repository and then click Next
Provide the following details in the Select and configure the action to perform screen
1. Select the Trigger generic build option
2. Provide the Jenkins base URL in Http://{ip address}:8080 format
3. Provide the User name and User API Token to trigger the build. Note that the username and Token are the credentials of the Jenkins user. For Token use, the API Token generated earlier from Jenkins
4. Select the Build job you created in Jenkins.
Click the Test button to validate the configuration and then click Finish. This will set the trigger to initiate the Jenkins CI build whenever a source code change is committed on the repository.
Try making a commit to the code - src/main/webapp/index.jsp would be a good candidate. This should trigger the MyShuttle build on Jenkins. You can confirm it by checking the history tab of the Jenkins services hook.

Approach 2: Wrapping Jenkins Job within Azure Pipelines

In this approach, Jenkins CI job will be nested within an Azure CI pipeline. The key benefit of this approach is that you can have end-to-end traceability from work items to source code to build and release pipelines.

To begin, an endpoint to the Jenkins Server for communication with Azure DevOps will be configured.

Go to your project settings. Select Pipelines and Service connections, click New service connection and choose Jenkins from the dropdown.
Provide a connection name, Jenkins server URL in the format http://[server IP address]:8080 and Jenkins user name with password (Use Jenkins User API Token as password). Select Verify Connection and validate the configuration. If it is successful, then select Ok.

The next step would be to configure the build pipeline.
Go to Pipelines | Pipelines, Click New Pipeline to create a new build definition.
Select Use the classic editor to create a pipeline without a YAML.
Select Myshuttle project, repository and click Continue.
Scroll down and select the standard Jenkins template and then click on Apply.
Select windows-2019 for the Agent specification, provide MyShuttle as the Job name ( name of the build definition that was created in Jenkins) and then select the Jenkins service endpoint created earlier.
Next, select the Get Sources step. Since Jenkins is being used for the build, there is no need to download the source code to the build agent. To skip syncing with the agent, select Don’t sync sources option.
Next, select the Queue Jenkins Job step. This task queues the job on the Jenkins server. Make sure that the services endpoint and the job names are correct. The Capture console output and the Capture pipeline output options available at this step will be selected.

The Capture console output and wait for completion option, when selected, will capture the output of the Jenkins build console when the Azure build pipeline runs. The build will wait until the Jenkins Job is completed. The Capture pipeline output and wait for pipeline completion option is very similar but applies to Jenkins pipelines (a build that has more than one job nested together).
The Jenkins Download Artifacts task will download the build artifacts from the Jenkins job to the staging directory.
The Publish Artifact drop will publish to Azure Pipelines.
Click on Save & queue button to save and initiate a new build.

Linking the build artifact for deployment in a CD pipeline

Next, you will configure an Azure CD pipeline to fetch and deploy the artifacts produced by the build. Since the deployment is being done to Azure, an endpoint to Azure will be configured. An endpoint to Jenkins server will also be configured, if not configured earlier.

After the endpoint creation, go to the Releases tab in Azure Pipelines. Click +New and select New release pipeline to create a new release pipeline.
Select the Azure App Service Deployment template.
The default environment for deployment will be named as Dev
In the Artifacts section in the Pipeline tab, choose the + Add link to select your build artifact.
1. If you have used the first approach, select Jenkins as the Source type, select the Jenkins endpoint configured earlier and provide MyShuttle for the Source(Job), choose the Default version as Latest. The Source(Job) should map to the project name configured in Jenkins.
  
  If the Jenkins server and the source location is configured correctly, once the publishing of the artifacts is completed, a message with the output file name myshuttledev.war will be displayed.
2. Otherwise, point this to the Azure CI build pipeline from which the Jenkins CI is executed.
Now, the artifact is linked for deployment. Please refer the Deploying a MySQL Backed Tomcat app on Azure Web App lab for deploying the WAR file to Azure App Service.