Integrating Xray with JMeter functional tests

This article describes an approach for importing results from JMeter functional tests in JIRA based test management tool – Xray by utilizing Jenkins CI platform.

Introduction

Xray is a popular test management tool add-on for JIRA. Xray supports manual and automated tests and a complete testing lifecycle: test planning, test designing and test execution. Xray has native support for Cucumber tests and also support for importing the test results from other testing frameworks like Junit, TestNG, NUnit, Robot Framework, Behave, Calabash and etc. However, Xray doesn’t have built-in support for importing JTL files that contains test results generated by JMeter, and as a solution for this we are proposing converting JTL files in a JUnit format as part of the ANT script that executes JMeter functional tests.

Ant script for JMeter tests run

In order to execute JMeter tests from Jenkins in our example, we will use ANT script. Below you can see a description of a used ANT script.

<path id="ant.jmeter.classpath">
	<fileset dir="C:\apache-ant-1.9.9\lib" >
	<include name="ant-jmeter-1.1.1.jar"/>
	</fileset>
</path>

Define a class path with the location of the “ant-jmeter-1.1.1.jar”:

<path id="ant.jmeter.classpath">
	<fileset dir="C:\apache-ant-1.9.9\lib" >
	<include name="ant-jmeter-1.1.1.jar"/>
	</fileset>
</path>

Create a jmeter ant task that will point to the class “JMeterTask”:


<taskdef name="jmeter" classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask" classpathref="ant.jmeter.classpath" />

Clean the directories with the results and the log file and create new directories for storing the new

results:

 specify JVM arguments to the JVM launched to run Jmeter.
<target name="test" depends="clean">
    <jmeter jmeterhome="${jmeter.home}" resultlogdir="results/jtl">
	<testplans dir="${basedir}" includes="CreateOrder.jmx"/>
	<property name="jmeter.save.saveservice.output_format" value="xml"/>
	<jvmarg value="-Xmx1536m"/>
    </jmeter>
</target>

Execute single or multiple JMeter tests by using the “<testplans>” tag, set the format and the output directory for the results. You can also specify JVM arguments to the JVM launched to run Jmeter.

<target name="report" depends="test">
    <xslt basedir="results/jtl" destdir="results/xml" includes="*.jtl" extension=".xml style="${basedir}\junit.xsl" />
</target>

Generate junit-xml report from the JTL results file by using XSLT transformation.

<target name="report" depends="test">
    <xslt basedir="results/jtl" destdir="results/xml" includes="*.jtl" extension=".xml style="${basedir}\junit.xsl" />
</target>

Because Xray has no built-in support for JMeter reports in .jtl format, the results should be converted to “junit-xml” format with XSLT in “junit.xsl” file:

<target name="test" depends="clean">
    <jmeter jmeterhome="${jmeter.home}" resultlogdir="results/jtl">
	<testplans dir="${basedir}" includes="CreateOrder.jmx"/>
	<property name="jmeter.save.saveservice.output_format" value="xml"/>
	<jvmarg value="-Xmx1536m"/>
    </jmeter>
</target>

Generate junit-xml report from the JTL results file by using XSLT transformation.
<target name="report" depends="test">
    <xslt basedir="results/jtl" destdir="results/xml" includes="*.jtl" extension=".xml style="${basedir}\junit.xsl" />
</target>

Because Xray has no built-in support for JMeter reports in .jtl format, the results need to be converted to “junit-xml” format with XSLT in “junit.xsl” file:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
	<xsl:output method="xml" indent="yes" encoding="UTF-8"/>
	<xsl:template match="/testResults">
		<testsuites>
		     <testsuite>
			<xsl:for-each select="*">
			       <testcase>
				<xsl:attribute name="classname"><xsl:value-of select="name()"/></xsl:attribute>
				<xsl:attribute name="name"><xsl:value-of select="@lb"/></xsl:attribute>
				<xsl:attribute name="time"><xsl:value-of select="@lt div 1000"/></xsl:attribute>
				<xsl:if test="assertionResult/failureMessage">
					<failure><xsl:value-of select="assertionResult/failureMessage"/></failure>
				</xsl:if>
				<xsl:if test="@s = 'false'">
					<xsl:if test="responseData">
					    <error><xsl:value-of select="responseData"/></error>
							</xsl:if>
						</xsl:if>
					</testcase>
				</xsl:for-each>
			</testsuite>
		</testsuites>
	</xsl:template>
</xsl:stylesheet>

Xray configuration

For the purpose of this demonstration, let’s consider that in Xray a Test Set named as “Create Order” that contains Xray Tests of type Generic is created . For more info on the specific issue types introduced by Xray, please refer to: https://confluence.xpand-it.com/display/XRAY/Terms+and+Concepts).