Integration of Azure DevOps with Xray Cloud

1. Introduction

As a testing tool, Xray easily integrates with other tools, whether we are implementing continuous integration on our system or we simply want to obtain some data. Since quality assurance is important, Xray supports the most common Test Runner reports when importing execution results like Cucumber, Junit, TestNG, NUnit.

If the Test cases being imported are not yet created in Xray/Jira, then Xray will create them automatically. The created tests will have the Generic test type and will contain the identifier or the name in the definition field. Xray will reuse the same Test issues in JIRA for subsequent imports of execution results. DevOps is the fast software development and deployment to enable continuous delivery of value to end-users by achieving incremental software delivery.

Azure DevOps is where teams manage, develop, and deliver software in the cloud. Azure DevOps provides free, private Git project hosting for up to 5 users, project and test management tools, build and CI, and team collaboration. It can be integrated with more than 2 000 Apps including Trello, Slack, GitHub, Jira Software Server/Cloud, Google Sheets, Gmail, and a lot more. There are also ready-to-use integrations that are available on the Internet but that’s not our focus in this blog post.

In the following blog post, we will use Azure DevOps as one CI tool that can interact with the Xray Jira project by executing tests from and send results back to the Xray issue. When the code is committed, it automatically builds the test project, enabling bugs detection early. This, in turn, will enable business organizations to achieve fast and identical deployment to the production environment at any given time.

2. Azure DevOps with Xray Cloud Integration

2.1 Pre-settings in Xray

Xray supports integration with Continuous Integration tools, thanks to its REST API. For this sample we have a cloud instance of Jira set up with the Xray plugin. First, we create one issue with ‘Test’ type and we add Gherkin reference in the Scenario section and for Test Details, we choose Cucumber as in Figure 1:

Cucumber feature files can be exported from Xray UI or through the command line and executed externally. They can be generated also from other Xray issue types, like Test Set, Test Execution, and Test Plan. Here is the curl command for exporting the features for the above issue:

curl -H “Content-Type: application/json” –output features.zip -X GET -H “Authorization: Bearer ${token}” “https://xray.cloud.xpand-it.com/api/v1/export/cucumber?keys= AZMARIJA-1″

The following feature file is exported from the sample above:

Feature: SignIn - Cucumber Test

                @TEST_AZMARIJA-1 @demo-desk
                Scenario: SignIn - Cucumber Test
                                Given User is on Home Page
                                When User Navigate to SignIn Page
                                And User enters valid UserName and valid Password
                                Then User should be successfully loged in and see MyAccount
                                Then End test by closing the window

2.2 Azure DevOps settings

In order to build a project in Azure DevOps, we need to know the structure and CI or pipeline executions. Because the purpose of this document is QA-oriented, not DevOps, we will stick just to the execution pipeline. It can execute whatever test with whatever tool we want but in this sample, we will use the Cucumber test tool as one of the default test kinds in Xray executions. The process is more or less the same for Junit, NUnit, or other testing tools.

First, to use Azure DevOps cloud services, we can use a Microsoft Account and sign up here. Then we need to build a new Project by clicking the New Project button and the newly created project will have the following route “dev.azure.com/{YourProjectName}”.

Because we will not go into details into Azure DevOps possibilities, we will use the Repo service which is an integrated repository of our project, and Pipelines where we execute the commands:

A Git repository, or repo, is a folder where the test code is located and Git tracks file changes in. When we click Repos on the left menu we can create the Git track in our folder that contains the maven project of the cucumber test (Figure 3). We can use other repo types as GitHub, Bucket, etc. but because Azure Repos service includes unlimited cloud-hosted private Git repository for our project, we will use the one that serves Azure DevOps.

A Git repo contains every version of every file saved in the repository location. Git saves these files very efficiently, so having a large number of versions doesn’t mean that it uses a lot of disk space. Storing each version of files helps Git merge code better and makes working with multiple versions of code quick and easy.

In the picture, we can see one additional file named azure-pipelines.yml. This is the main file for building the pipeline. Azure Pipeline is a cloud-hosted pipeline for fast CI/CD that works with any language, platform, and cloud. YAML (Yet Another Markup Language) files are very useful in writing build and release definitions. Azure Pipelines has components like build, release, library, task groups, and deployment groups. The azure-pipeline.yml consists of the code for extracting the features file/s from Xray, building (running the test), and sending the results back by creating a Test Execution in Xray issue.

trigger:
- master
 
pool:
  vmImage: 'windows-2019'
 
steps:
# With this bash command we interact with the Xray Jira project and we pull the feature/features file from the required Xray - cucumber issue

- bash: |
    token=$(curl -H "Content-Type: application/json" -X POST --data "{ \"client_id\": \"$(client_id)\",\"client_secret\": \"$(client_secret)\" }" https://xray.cloud.xpand-it.com/api/v1/authenticate| tr -d '"')
    echo "##vso[task.setvariable variable=token;]$token"
  displayName: GetToken

# We request Xray REST API to get the feature file from the specified issue
- bash: |
    curl -H "Content-Type: application/json" --output features/features.zip -X GET -H "Authorization: Bearer $(token)"  "https://xray.cloud.xpand-it.com/api/v1/export/cucumber?keys=$(keys)"
    unzip -o features/features.zip -d ./IWECProject/features
  displayName: 'Get Feature From Xray'

# This step is the maven build of the project - it depends how our project is built and how we want to be run
- task: Maven@3
  inputs:
   mavenPomFile: 'IWECProject/pom.xml'
   mavenOptions: '-Xmx3072m'
   javaHomeOption: 'JDKVersion'
   jdkVersionOption: '1.11'
   jdkArchitectureOption: 'x64'
   publishJUnitResults: false
   testResultsFiles: '**/*.json'
   goals: 'clean compile test'
   
# With this bash command we push cucumber.json results file (created from the run in the before step) in the required Xray - cucumber issue

- bash: |
    curl -H "Content-Type: application/json" -X POST -H "Authorization: Bearer $(token)" --data @IWECProject/cucumber.json "https://xray.cloud.xpand-it.com/api/v1/import/execution/cucumber"
    echo "Sending results to XRay done"
  displayName: 'Send Results To Xray'
 
# Generate HTML report to display it in Azure Dashboard
- bash: |
    npm i cucumber-html-reporter
    node index.js
  displayName: 'Generate HTML Report'
  
# Publish HTML report in Azure DevOps 
- task: PublishHtmlReport@1
  inputs:
    reportDir: 'D:\a\1\s\cucumber_report.html'

As we can see from the yml file we use REST API calls for interaction with Xray. The client_id, client_secret, and keys are set up as variables for the pipeline in the settings of the pipeline. Those parameters are needed for authorization and interaction with the Jira instance. For security reasons, we can set client_id and client_secret variables as secret values by checking the option “Keep this value secret”. Also, variables are used for easy change of the value, for example, the keys variable is the name for the issue that we want to interact with and probably we want to change it frequently. That is how we will be able to run the same pipeline for different or multiple issues.

The cucumber.json result output file (–data @IWECProject/target/cucumber.json), generated by the Test runner class, is used to be sent back to the Xray issue and will create Test Execution type issue with the automatically generated name “Execution results $timestamp” and will be related to the Cucumber Test that we pulled the feature file (“AZMARIJA-1”) as on Image 5.

Conclusion

Xray for Jira project management tool is a highly recommended plugin, because of the easy usage and possibility for integration with a lot of applications/tools. One of the abilities of Xray is integration with Azure DevOps and its Continuous Integration tools, thanks to the Xray REST API. With such integration, we can track all the changes, modifications, or regressions in one place. By creating an Xray issue, for example with Gherkin kind reference, we can automatically run and have an execution status updated with minimal or no human effort. The automation process is done by the Azure DevOps pipeline by pulling the features, executing the test, and sending the results back to the Xray – by creating a Test Execution for the related issue.

If you want to discuss the integration of Azure DevOps with Xray Cloud, you can contact us at sales@iwconnect.com.