Screenshot Testing for Android to Discover UI Regressions

_{by Markus Hintersteiner, Software Architect Android}

Testing on Android has come a long way over the past few years. Nowadays tools like Mockito and Espresso make it easy to write unit and UI tests for Android. At Runtastic, our Android engineers share a lot of UI components across our apps. With screenshot testing, we can test our UI components against visual regressions under various conditions. In this blog post, we’ll show you how that works and how you can apply screenshot testing for your projects too.

How does screenshot testing work in general?

Screenshot testing uses instrumented UI tests to render a screen or a portion of a screen to an image. This image is then compared to a “reference image”. If both images are equal, the test succeeds – otherwise it fails.

A screenshot test is quite similar to a UI test. The only difference is that for assertion the rendered image is compared to a reference image.

Introducing the screenshot testing library

At the moment, the Android Testing Support Library does not offer any tool to perform screenshot tests. Luckily, Facebook published a screenshot-tests-for-android library a while ago. One of the nice things about the library is that it allows you to screenshot test Android UI widgets without embedding them into an activity, thus allowing you to truly stand-alone test your widget.

In order to meet our requirements, we forked the library and extended the feature set. It’s available under: https://github.com/runtastic/screenshot-tests-for-android

Our fork extends the existing tool with the following functionality:

• Gradle API improvements
• Automation of storage runtime permission handling for Android M and above
• JUnit test report XML generation suitable for CI integration
• Side-by-side HTML report of your screenshot tests

The screenshot testing tool consists of two parts:

1. A core plugin which provides an API for your test code to capture screenshots. When running your tests, the screenshots get stored on the external storage of your device/emulator.

2. A Gradle plugin which is responsible for pulling the screenshots from the external storage and performing further processing depending on the mode.

The screenshot testing tool offers two modes: record and verify

1. The record mode just pulls the screenshots into the screenshots folder of your module. This folder is then ready to get checked into your version control system.

2. The verify mode compares the pulled images with the previous images from the screenshots folder. If two images do not match, the Gradle task will throw an error indicating that your screenshot test has failed.

Execute your first screenshot test

Prepare your environment

The screenshot-tests-for-android tool uses Python 2.7 and a few Python modules which can be installed using pip.

On a Debian-based Linux distribution, you could install them using the following commands:

sudo apt-get install python python-pip
pip install --user pillow mako junit-xml

Our fork is currently not available via Maven, but you can easily install the artifacts locally:

git clone https://github.com/runtastic/screenshot-tests-for-android
cd screenshot-tests-for-android
git checkout feature/v0.4.3
./gradlew installArchives

Prepare your Android project

buildscript {
    dependencies {
        classpath 'com.facebook.testing.screenshot:plugin:0.4.3-SNAPSHOT'
    }
}

In your module build.gradle, apply the testing plugin: 

apply plugin: 'com.facebook.testing.screenshot'

Prepare your testing flavor

Because the recorded screenshots are stored on the external storage, you need to add the appropriate manifest permissions to your androidTest flavor.

src/androidTest/AndroidManifest.xml:

<?xml version="1.0" encoding="utf-8"?>
<manifest
    package="com.runtastic.android.ui.lego"
    xmlns:android="http://schemas.android.com/apk/res/android">

    <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
</manifest>

Write your first screenshot test

Now it’s time to write your first test! Here’s a basic example of how a test could look like:

    import com.facebook.testing.screenshot.Screenshot;
    import com.facebook.testing.screenshot.ViewHelpers;

    public class RtProgressBarTest extends InstrumentationTestCase {

        @Test
        public void solidStyle0p() throws Exception {
            final Context context = getThemedContext(R.style.Theme_Runtastic);

            final RtProgressBar progressBar = new RtProgressBar(context);
            progressBar.setProgress(0.0f);

            ViewHelpers.setupView(progressBar)
                    .setExactHeightDp(16)
                    .setExactWidthDp(240)
                    .layout();

            Screenshot.snap(progressBar)
                .setName("progress_solid_0")
                .record();
        }
    }

In the test above, we create our RtProgressBar widget and set the progress to 0%. Then we use the ViewHelpers class to measure and layout the widget. Finally, the Screenshot class is used to record an image.

As you can see, there is no need to perform any assertions. This is done automatically by the Gradle plugin for every recorded image.

Run your screenshot test

As mentioned earlier, there are two modes: record and verify.

In record mode, the screenshots are saved as a reference and will be later used for verification.

./gradlew recordScreenshotTests

After the Gradle task finishes, you should see a screenshots folder in your module directory. The contents should be treated as part of your source code and only change when you add/remove tests or change the UI of existing components.

After a code change which does not affect your UI, you typically run the verify task to make sure there is no visual regression.

./gradlew verifyScreenshotTests

To review the screenshot test results navigate to:
file:///path/to/module/build/reports/screenshots/index.html

If your verification succeeds, the Gradle task finishes successfully; otherwise, it fails. The Gradle output shows you which tests failed.

Additionally, two reports are generated:

• a JUnit test report under build/reports/screenshots/screenshot_report.xml
• a HTML report under build/reports/screenshots/index.html

The HTML report allows you to easily check for regressions. Here’s a screenshot:

Wrap-up

I hope you enjoyed this quick intro into screenshot testing. Be sure to check out the official documentation http://facebook.github.io/screenshot-tests-for-android/ for more details.

Feel free to add a comment below!

***