Configure AWS Device Farm for running Java Appium mobile automated tests

14 Jul, 2017 | 3 minutes read

Introduction

Device Farm is an app testing service that enables you to test and interact as an AWS engineer with your Android, iOS, and Web apps on real, physical phones and tablets that are hosted by Amazon Web Services (AWS). There are two main ways to use Device Farm:

  • Automated testing of apps using a variety of available testing frameworks
  • Remote access of devices onto which you can load, run, and interact with apps in real time

Device Farm allows you to upload your own tests or use built-in, script-free compatibility tests. Because testing is automatically performed in parallel, tests on multiple devices begin in minutes.


A test report containing high-level results, low-level logs, pixel-to-pixel screenshots, and performance data is updated as tests are completed.


Device Farm supports testing of native and hybrid Android, iOS, and Fire OS apps, including those created with PhoneGap, Titanium, Xamarin, Unity, and other frameworks. It also supports remote access to Android apps for interactive testing.

Device Farm currently provides support for the following test types:

Device Farm and the support it provides

Note: For the development of the automated tests, we used the Appium JUnit framework for the “Calculator” application on Android devices.

Prepare Android Appium Java JUnit Tests for running on AWS Device Farm

Android Appium Java JUnit tests must be contained in a .zip file. The POM file needs to be set up according to the official AWS Device Farm documentation for JUnit.

The following dependencies should be added to the maven POM file:

<p class="has-text-align-left"><br><script></script></p>

Go into your Appium Maven project directory from the command prompt and run the following command to package the test content.
mvn clean package -DskipTests=true

After execution of this command, the “zip-with-dependencies.zip” file will be created in the target folder, which will contain .jar files of the test classes and all dependency .jar files.

Zip with dependencies file

Note: The “zip-with-dependencies.zip” file needs to be uploaded when creating a run on AWS Device Farm.

Create a test run on AWS Device Farm

  1. Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm
AWS device farm interface after logging in

2. Click on “Create a new project”, enter a project name, and click on the “Create project” button.

Create a project interface

3. Click on the “Create a new run” button on the project page.

Create a new run

4. On the “Choose your application” page, choose either Native application (the Android and Apple button) or Web application (the HTML5 button). Upload your application file and click on the “Next step” button.

Choose your application” page, choose either Native application (the Android and Apple button) or Web application (the HTML5 button)

5. On Configure a test page, select test type, ex. “Appium Java JUnit”, upload the “zip-with-dependencies.zip” file. It will take a few minutes for the tests to upload, after the test upload is finished select “Appium version” and click on the “Next step” button.

Configure a test

6. Select the device (ex. Huawei P9) on “Select devices” and click on “Next step” button.

Select the device

7. Click on the “Next step” button on the “Specify device state” page.

8. On “Review and start run” page set the execution timeout per device “ex. 10 minutes” and click on “Confirm and start run”.

Review and start run

9. A report in Device Farm contains information about a run, which is a request for Device Farm to test a single app against one or more devices. Reports in Device Farm contain pass and fail information, crash reports, test and device logs, screenshots, and performance data. Reports include both detailed per-device data as well as high-level results, such as the number of occurrences of a given problem.

9.1 The “Video” section displays a downloadable video recording of the test.

a downloadable video recording of the test

9.2 The “Suites” section displays all test steps that contain pass and fail information and time of execution for each test step.

Test steps that contain pass and fail information and time of execution for each test step.

9.3 The “Logs” section displays any information Device Farm logged during the test.

Information logged during the test

9.4 The “Performance” section displays information about any performance data Device Farm generated during the test.

information about any performance data generated during the test

Conclusion

AWS Device Farm is easy to use and offers a very detailed view of the results, providing all the information needed to reproduce the bug. The main advantage is the possibility to execute tests on a wide range of devices, allowing us to discover problems, which can’t be seen on the local test devices.