Xray is a complete Test Management tool for Jira. It is a full-featured app that does not require any other software in order to run. Xray supports the entire testing life cycle: test planning, test design, test execution and test reporting.
On the other side, Cucumber is a framework for behavior-driven development (BDD), where the test specification is done in a natural language, while the implementation of each specification’s phrase (i.e. step) is done in code (e.g. Java, Ruby or some other language).
In this article we give an explanation of the steps about how you can create Automated Cucumber test with the help of Xray and how you can import the automated test results back in JIRA.
Create Cucumber Test with Xray
One of the ways to create an automated Cucumber using Xray is to navigate to the Jira user story that you are planning to test.
Figure 1. JIRA User Story
By clicking on Create new Test button, the form for creating a test case is opened. In order to create Cucumber Xray test, under Test Details tab, for Test Type choose Cucumber and from Cucumber Type drop-down choose Scenario. (The option Scenario Outline is used when you want to create instructions that contain variables).
Figure 2. Cucumber Test Creation with Xray
The next step is to populate Cucumber Scenario with test steps in Gherkin language.
Figure 3. Cucumber Test Creation – Creating test steps in Gherkin Language
Under the Link issues tab you can see that the test is automatically linked to the user story from where the test was created.
Figure 4. Cucumber Test Creation – Associate test to User Story
By clicking on “Create” button the Cucumber test will be created.
Figure 5. Xray Cucumber Test in JIRA
Test Execution and Importing Test Results in JIRA
Xray enables exporting Cucumber tests to be exported as .feature file that further can be used in your automated tests. In order to export .feature file, from the created test choose More -> Export to Cucumber.
Figure 6. Exporting .feature file from Xray Cucumber Test
Further, you can import the exported .feature file in your test automation project, for example implemented in Eclipse.
Figure 7. Exported .feature file in Eclipse
After implementing the step definition file, using for example Selenium Web Driver, for the given feature file, and executing Cucumber test in Eclipse, Cucumber.json file with results will be generated under the target folder in your Eclipse project.
Figure 8. Cucumber.json file with test result after test execution in Eclipse
Using Xray, the test results from the generated json file with the test results can be imported back in JIRA, and associated with the corresponding test, by manually importing the json file or by using API that is provided by Xray.
Importing Cucumber automated test results in JIRA through UI
In order to import the generated result from the automated tests, in the Test Execution* associated with the Cucumber test, click on More -> Import Execution Results and import the generated json file in JIRA. (* Test Execution is custom JIRA issue type introduced by Xray, that can be described as an assignable, “schedulable” task to execute one or more tests for a given version/revision along with its results. For more information on different Xray custom JIRA issue types, please refer to https://confluence.xpand-addons.com/display/XRAY/Terms+and+Concepts)
Figure 9. Import Cucumber.json file with test results in Test Execution
Once the execution results are imported in JIRA, Test status is updated. In our case the test has failed, so correspondingly its status is updated in the associated Test Execution.
Figure 10. Updated Test status after importing .json with test results
Execution details, including the error that was thrown during automated test execution, can be accessed as depicted on Figure. 10
Figure 11. Test Execution Details
Importing Cucumber automated test results in JIRA through API
Xray provides specific REST endpoints for different test execution results depending on the testing framework that is used for test automation. In the table below are listed some of the REST endpoints, while the full list can be seen on: https://confluence.xpand-addons.com/display/XRAY/Import+Execution+Results+-+REST
In a real-world scenario, importing automated tests results in JIRA through Xray provided APIs will be executed in conjunction with CI/CD tools, like Jenkins, however in this article, for demonstration purposes we will use JMeter to perform POST operation on the REST endpoint for Cucumber test result output format: /rest/raven/1.0/import/execution/cucumber
Figure 12. JMeter script for importing results through Xray REST API
After invoking REST API, through JMeter, corresponding Test Execution associated with the Test is updated, showing Test as Pass:
Figure 13. Test execution results in JIRA after importing through API
Xray plug-in for JIRA provides ideal solution for test management on all projects where JIRA is used for tracking requirements and bugs. Besides it supports management of test cases, test planning, test execution and requirements coverage with its built-in support of Cucumber and other test automation frameworks enabling support whether is needed manual or automate testing.