An API is a set of programming instructions for accessing a web-based software application, and QA engineers often use testing to compare the response with the expected results. In fact, it is a set of commands used by an individual program to communicate with one another directly and use each other’s functions to get information.
REST is an architectural style and an approach for communication used in the development of Web Services. It enables users to connect and interact with cloud services efficiently. In other words, when you type a URL in a browser, you’re sending a request to the web server and the web server responds with a resource.
When it comes to testing REST APIs, we will consider them properly tested when the following areas are covered:
- Verifying that API service is properly built based on REST standards
- Verifying response status codes, response body, and response header
- Assertion of response content
API testing requires specific tools, dedicated to automating it. The choice varies depending on how they fit in an existing REST API project.
Apache JMeter and Rest Assured are two well-known REST tools that can be used to test REST API services. My main focus here is to compare their functionalities and point advantages and disadvantages over each other.
What is JMeter?
- JMeter is a free open-source Java software, used for functional API testing although it is primarily designed to load test functional behavior and measure performance.
- However, JMeter has all the needed functionalities for API testing and have also some extra features such as automatically working with CSV files which means coding is almost not required. It can be used to create test fragments that can be reused in other suites.
- JMeter resembles a browser or multiple browsers but it does not support all browser actions.
What is Rest Assured?
- REST-Assured is an API testing tool, which is considered to be the most established and with a powerful Java-based API that leverages Hamcrest as the matcher tool. It also has a simple set of methods that allows method chaining and Pseudo-Gherkin Syntax.
- Actually, REST-Assured is a Java library supporting XML and JSON requests, you can use it to test HTTP-based REST services and it integrates with any existing Java-based automation framework. This means that you have one framework which allows you to combine your UI and REST tests and it also generates awesome reports. It supports GET, POST, PUT, DELETE, HEAD, OPTIONS, and PATCH requests and can be used to validate as well as verify the response of all these requests.
Comparison of JMeter vs Rest Assured
1. As per installation and setup, JMeter and Rest Assured require installed Java but:
a) JMeter installation is very easy, just download the archive, unpack, and install it. The tool is ready to use.
b) Rest assured is not a standalone tool and requires already installed Eclipse and Maven. Since it is a Java library, in order to use it, a new Java project has to be created, and the Rest Assured library is included in it. “pom.xml” file should be modified properly and appropriate dependencies included.
2. Documentation:
a) JMeter documentation can be found on its official website where links are provided to relevant rich resources like video tutorials and so on.
b) Rest Assured documentation can be found in the GitHub repository where detailed explanations of “how-to” are provided.
3. Regarding automated test creation:
a) JMeter has GUI and test creation is from the GUI, however, that does not limit possibilities regarding test planning and creation. The BeanShell component allows adding some custom “conditions” in the test case.
b) Rest Assured doesn’t have a user-friendly GUI and tests are created using Java coding skills. It has a lot of already-created functionalities, greatly easing the job as you don’t have to code things from scratch.
4. Collaboration with others on the same project:
a) In JMeter, a project is saved as a huge XML file, and even for the smallest change in some tests, the whole project must be re-opened in the GUI. In cases when a merge conflict occurs during collaboration, it could be difficult to fix/resolve it. JMeter doesn’t have an option to resolve conflict directly from GUI.
b) Rest Assured is very easy and simple to maintain using any SCM system. In order to change something, you will need to open just one (appropriate) file.
5. Functionality:
a) JMeter is a powerful performance test tool and supports very interesting reporting options including different charts and tables. JMeter supports SOAP/REST API performance, functional, and regression testing.
b) Rest Assured has limited functionalities but is still powerful. Its primary usage is functional REST API testing. It has its own response assertion mechanism. But If you would like to use it as a performance testing tool you will probably have to develop your own custom framework.
6. Execution:
a) JMeter has its own run button in the GUI. It can be run also from the command line.
jmeter -n -t your_script_name.jmx
b) Rest Assured requires a separate runner, like TestNG. It can be run also from the command line. If you use TestNG, dependency should be added in pom.xml file.
7. Reports:
a) Since JMeter is initially a performance testing tool, it supports a huge number of reports types itself. One of the most important things is that JMeter supports dashboard report generation (at the end of a run or on-demand). This characteristic enables the user to get graphs and statistics from a test plan.
b) When it comes to reports, Rest Assured does not have the option of creating reports by itself. But if it is integrated with some testing framework, you will get pretty good reports. If you are using the Test NG framework, the report does give good information (like the number of test cases passed, failed, and skipped, and total execution time with a number of groups excluded or included.) but if Extent is added then the report will be very useful and more powerful.
8. When it comes to CI/CD integration, both can be easily integrated for example with Jenkins since it has a plugin for almost all testing frameworks.
9. Releases and community activity:
a) JMeter has a large community mainly because it has been out in the market for a longer time as compared to Rest Assured. An official release is being published every 5-6 months. Since the most recent release in November 2019, JMeter 5.2 has been packed with multiple great features and enhancements, the user experience has been improved, and many bugs were fixed, such as new protocol, JMESPath extractor, JDBC improvements, StringtoFile, HTTP Samplers.
b) Rest Assured is making its way to becoming the primary choice when it comes to testing RESTful services. Similar to JMeter, Rest Assure publishes official releases every 6 months. Starting from version 4.0.0, REST Assured requires at least Java 8, instead of Java 6 as was the case previously. This version also added support for Apache Johnson and fixed a lot of issues with the initial OSGi support. The latest version 4.2.0 announced in January 2020 allowed a nicer experience for Kotlin developers using the spring-mock-mvc module.
Summary
To sum up, JMeter and Rest Assured are very robust, with nice and useful testing frameworks, but still have differences.
| JMeter | Rest Assured | |
| Coding skills | No | Yes |
| Easy setting up | Yes | No |
| Documentation | Yes | Yes |
| Functional testing | Yes | Yes |
| Performance testing | Yes | No (unless you create custom framework) |
| Collaboration through SCM system | No | Yes |
| Reports | Yes | No (Plugin) |
| CI/CD Integration | Yes | Yes |
| Unit Tests Integration | No | Yes |
| Parallel testing | Yes | Yes |
| Community | Yes | Yes |
| Official releases | Yes, every 5-6 months | Yes, every 6 months |
| Price | Open-source | Open-source |
While there are minimum coding skills required in JMeter, in Rest Assure you need to have coding skills to write even a simple test. JMeter provides good reports, while Rest Assure needs a plugin for that purpose.
Conclusion
Finding the perfect tool is tough. Besides all of the aforementioned strengths and weaknesses of JMeter and Rest Assure, the ready-made tool JMeter would be the choice that suits best due to its better usability, interaction, and extra features:
- Created API functional tests can be leveraged in performance tests. So duplicating REST API calls can be simply avoided.
- JMeter works automatically with CSV files allowing quick creation of unique parameter values. It also executes tests fast and provides accurate timing. Out of the box, it builds comprehensive, meaningful reports showing performance and functional metrics.
- Community! JMeter has a larger community mostly because it is no longer in the market. Its Github repository is very active on a daily basis.
