Screenshot testing on Android
Screenshot testing on Android
I just want to sleep well the night after we release a new version of Jobandtalent’s android app. In the first place, here, at jobandtalent Engineering, we have around 1,3K unit tests. When talking about testing our user interface, we also have quite a large suite of UI tests (another set of more than 1K instrumentation tests) covering the different scenarios that our users may come across.
We use Espresso to verify that our app responds and behaves as expected. However, there are some cases in which Espresso does not fit properly into what we want to achieve. We need a more powerful tool for those concrete scenarios. And this is why we began our experiment adopting an additional approach to UI testing: screenshot testing.
In the following article, I am just presenting a strategy that works fine for us. This is neither universal truth nor something that may work for every team.
Our motivation for dealing with testing
There are two basic things that we want to make sure that our code is doing: testing user inputs and visual outputs to the user.
Testing user inputs
On the one hand, we may want to check that the app manages correctly a concrete user input. For instance, when our users tap the login button, we want our login presenter to be notified about this event, so it executes the underlying logic. This is where we make use of Espresso to perform those necessary assertions on the code.
In the previous example, we are checking that method onRefreshClick() in our presenter is actually called when the user taps the button with the text R.string.common_retry.button.
Testing visual outputs
On the other hand, we may want to test that the UI updates correctly to provide the user with visual output. For example, when there is a network error, we want the user to see an error state explaining the problem. In this case you can, of course, do something like this:
This works fine. This test will fail if either the title, message, picture or button of the error state is not shown. Also, it will fail in the case that the message text is incorrect. Now is when it comes to the decision. Is this enough for us? Actually, it may be enough for some scenarios and that’s OK. There is no point in testing every little thing if that doesn’t provide us with an income of some kind.
Let’s have a closer look at what that test is not testing.
Screenshot testing
In the previous example, we are just checking that the picture is being shown, but we are not making any assertion on what picture is being shown. Furthermore, the picture must be in the center of the screen, but it may actually be on the right because of a bug and that test will not fail either. Not to mention that paddings could be one pixel wider than they should, and I only would realize this when a designer tells me (sorry guys, my eyes are not that precise 😉).
You may think that this kind of bugs is not so common and that finding an alternative is not worth the effort. But we all know that upgrading the support library (or migrating to AndroidX) sometimes has unexpected outcomes. Wouldn’t it be great to have some automatic check that reports any problem with these small details that traditional instrumented tests are not covering?
So, to avoid these kinds of tricky bugs to be detected in production, at Jobandtalent we decided to implement screenshot testing.
Our implementation
In case you are not familiar with screenshot testing, I strongly suggest you have a look at the Facebook screenshot testing library. Screenshot testing basically consists of taking screenshots of some state of your app and comparing them against some reference that is considered to be the source truth.
If you have been curious about that link before, you may have noticed that it is a library Facebook has developed to make screenshot testing easier. We are making use of it to implement our screenshots, along with the shot Gradle plugin that our friends at Karumi have developed. This plugin makes a lot easier to read the output from the Facebook screenshot testing library. It generates an HTML file that shows the original screenshots recorded, the actual screenshots were taken and the differences for each one. This allows us to check the differences with a single glance.
The process that we have adopted consists of the following steps:
- In the first place, we implement our new custom view (or fragment/activity) and its corresponding Espresso tests as normal (screenshot and Espresso tests do not need to be exclusive, of course).
- Afterward, we think about those regular cases that the user will see (i.e the happy paths).
- We have to slightly change the way we create tests and now think about those stranger scenarios and how the app should handle them. For example, what happens if a
TextViewcontains a very long line of text (should it wrap onto several lines or should it just ellipsize after the first line?).
- Then we run the shot plugin task to record the references for the first time.
- Finally, we review the references, if possible along with a member of our design team, to check that the pictures reflect what we expect from the app. If everything is OK, we commit those references into our repository, converting them into the source of truth.
Following executions of the shot plugin will compare the screenshot results, pixel by pixel, against those references in the repository. These executions run automatically on our Jenkins environment, so if I have changed something in the UI that I was not supposed to change (even a color), then the corresponding test will break (and I will probably have to buy ice-cream for the rest of team…). When there is a failing test, the plugin will warn us and show us the difference from the reference UI.
As you can see, the difference specifies what’s actually the problem showing explicitly the difference:
This mechanism is especially useful for us when testing custom views that represent components of our design system (you may want to check out our post Working with a design system by my colleague Marcos Trujillo). Once we have recorded our references, we use them not only for testing automatically against them, but also to have a visual archive of the most common (and not so common) visual scenarios that a user may run into. However, we also have screenshot tests for whole activities and fragments.
Conclusion
Screenshot testing is a powerful tool that allows us to test pixel by pixel our user interface state. It has some clear advantages:
- It can detect bugs that will remain undetected using standard Espresso tests. For example bad layout implementation, incorrect spaces, paddings or colours…
- The design team can come and join us during this testing process. This allows us to receive early feedback from them, instead of showing them the final implementation or releasing an internal beta version. Feedback during this phase is easier to apply than feedback received, for example, during a QA session.
- These tests can also run automatically on our CI environment, so failures will be reported in a short period of time.
However, this does not come for free. It has some disadvantages that should be taken into account:
- Tests become more fragile. Any minimal desired change will make the references outdated, meaning that they need to be recorded again.
- Recording references is not a very fast process.
- These tests generate a lot of
pngfiles that need to be stored in the repository.
This system can be improved in the near future in several ways:
- Opening as many testing branches as languages supported by our app to make sure that everything is working fine for every locale configuration.
- Open new testing branches that use different API versions and devices, so we can check that some concrete view is working properly in different Android versions.
- Automatically publish the recorded references so they can be checked out not only by the design and engineering team but also by the rest of the company.
As a conclusion, we can say that this kind of tests has helped us a lot to detect bugs with much higher precision. However, if some views change very often, we have to rethink if all the processes that come with this testing mechanism are worthwhile. Like any other thing, this is the right tool for the right problem.
Kudos
I want to give some kudos to the Karumi team for that awesome formation on mobile testing received in one of their courses which I can’t recommend enough.
