The Power of Visual Testing with Applitools

Introduction

They say “A picture is worth a thousand words”, and Visual Testing proved this once again. It is a trend that has been in constant development in the last period, especially with the increased use of AI across organizations. But why is it getting so popular?

About Visual Testing

Visual Testing is a quality assurance activity that is meant to verify that the UI appears correctly to the users. Its image-based validation tool is built on top of AI algorithms and is used for automated visual validation. It helps find visual bugs that a typical functional regression test would not discover because the app might function as intended, but its UI can be a complete mess.

With visual testing, we can confirm that besides its function, the UI looks correct to the user and that each UI element appears in the right color, shape, position, size, and that it does not overlap any other UI element. It also verifies that the elements appear and function perfectly on a variety of devices and browsers.

This is why visual testing cannot be considered as a replacement of traditional functional testing but as an addition to it.

The examples above show some popular visual bugs that made their way to the production, just because the traditional regression test did not catch it. And no wonder, as the application behaved functionally correct, but no human eye checked the UI itself.

(Image source: Applitools.com)

Applitools Eyes SDK

Applitools Eyes is a cloud-based automated visual testing tool that tests the application like a manual tester would do – except that it works much more quickly and accurately. It can be integrated with many testing frameworks such as Selenium, Cypress, WebdriverIO, Selenium IDE, Testcafe, Protractor, etc, providing an easy way to include it in any already built tests.

With Applitools Ultrafast Grid, web applications can be tested on multiple browsers, in parallel, in as many screen resolutions as you want, on multiple mobile applications, which radically saves the time needed for test execution.

The system consists of the following major components:

• The application is under test.
• A test suite that exercises the application and verifies its correctness.
• The Eyes SDK, called in the test suite to capture the screenshot and do the visual testing.
• An application Driver, such as Selenium or Appium.
• The Eyes Server, which receives the screenshots from the SDK, stores them, makes comparisons, and reports differences.
• The Eyes Test Manager, allows users to review test results, report bugs, and manage the baseline images.

The way it works is quite simple. Once the application is run and it enters some state, a command is executed to capture a screenshot for each of those states. Eyes offers a variety of methods for making screenshots:

• checkWindow() – to capture only the visible screen
• checkWindow() with .setForceFullPageScreenshot(true) enabled – to force a capture of the entire screen, in case scrollbar is present.
• checkElement(locator) – to capture only the area of the provided element.
• checkFrame(locator) – to capture only the area of the provided frame.
• check() – Fluent API call that allows passing parameters for configuring special processing, such as target area, match level, regions to ignore, etc.

The saved screenshots are called checkpoints. Applitools keeps the screenshots on the cloud and is using them as baseline images, so whenever the application is re-run, the baseline image would be compared to the new screenshot.

The result of each re-run can be the following:

1. No differences. Meaning no changes in that part of the UI have been made. The test is marked as PASSED.

2. Differences found due to a new feature added that was previously not included in the baseline image. The new screenshot would be most likely accepted as the new baseline image and from this point, it will be used for comparison.

3. Differences found due to changes in UI that cause a bug in the application. The screenshot will be rejected, which will mark the test as FAILED. Furthermore, Applitools provides an option to integrate with Jira, so the issue can be created with all the details needed.

Another great option that is configurable in the Eyes Test Manager, is adding annotations to some regions of the image, to control the matching mechanism:

• Ignore: Don’t report any mismatches in this region.
• Floating: Allow the position of a region to move without reporting a mismatch.
• Match Levels:
  • Strict: Detect any mismatch visible to the human eye.
  • Content: Allow small variations such as ignoring color.
  • Layout: Check only the layout and ignore actual text and graphics.

Applitools Ultrafast Grid

Aplitools Ultrafast Grid as a service offers running the visual tests in parallel for:

• Different screen resolutions on multiple browsers,
• Emulated mobile devices,
• iOS mobile devices.

Unlike with classic eye runner, when a checkpoint image is made and it is saved on cloud, the ultrafast grid gathers all resources (HTML, CSS, fonts, images and so on), renders an image of the full page and sends it to the Eyes Server. With this it saves significant bandwidth and time for test executions, when cross-browser testing is needed.

Applitools Eyes in Action

In continuation, let’s see an example of Applitools Eyes integrated into the existing Selenium test, which was built based on the Page Object Model. The test was created to automate UI tests for a test e-commerce application: https://demo.nopcommerce.com/.

Eyes methods were used in test “GuestSearchesForProductAddsToWishlist”, which basically tests the following actions:

1. Guest navigates to the e-commerce application
2. Guest searches for a product (Adidas shoes)
3. Guest opens product details
4. Guest adds the product to wishlist

With a typical functional UI test, we would need a long list of assertions to validate that each element is present on the screen, for different screen resolutions. Even then, when all elements are present on the screen, we would still need a human eye to check that the elements are correctly presented. By doing a simple screen comparison with Applitools Eyes, we can save my time and lines of code.

Eyes is useful in this test to validate that search results are showing only the product that was actually searched in the search field.

Additionally, all those assertions on the Product Details – page, can be simply replaced with one line of code, by using the eyesManager.validateEntireScreen (“Product details”);

Once the product details are selected, Eyes was called once again, to validate that screen looks correct for the user.

Package com.commerce.basetests contains two classes:

• BaseTests.java
• EyesManager.java

Into BaseTests.java an eyesManager() method was added, which calls one of the constructors of the EyesManager class, depending on if the test should be run with ClassicRunner or with VisualGridRunner.

For the VisualGrid run, it is necessary to set configurations such as browsers, screen resolution, other devices on which the test should run.

The EyesManager.java class is created as a good practice to store all methods used for Aplitools Eyes.

(Image 13 – EyesManager.java class)

To be able to run the tests, an API key needs to be set, as well as an instance of the Eyes class. When instantiating the class, the runner should be also provided:

• ClassicRunner – for regular run on single device
• VisualGridRunner – for running the tests on multiple devices and screen resolutions

validateWindow(), validateEntireScreen() and validateElement() are methods that simply do the screenshot of the UI. eyesAwaitTestResults() is a method that communicates with the Eyes server, tells the test to wait for all tests to finish rendering on the Ultrafast Grid before ending the test suite. It should be called only once the tests are finished.

Conclusion

Visual testing with Applitools Eyes tool is an easy add-on to classical automated functional testing. They support a lot of the already known testing frameworks which makes it easier to integrate into existing tests. Their powerful Ultrafast Grid enables cross-browser and cross-mobile testing for nearly the same time needed to execute a single test.

If you are interested in more details, feel free to contact us!