Client Testing Strategy

2019 status: Current

Summary

The Buendia client has a suite of both unit and functional tests that should undergo continued refinement and growth. This document describes our testing methodology and where gaps still exist.

Testing Tools

Unit tests and functional tests share a common source tree in src/androidTest. There are a number of tools, utilities and base classes that make further test development simpler, including:

• Espresso -- used to run functional tests
• Mockito -- used to mock out objects that aren't under test
• Fakes
• FakeAppLocationTreeFactory (for constructing fake AppLocationTree objects)
• FakeAsyncTaskRunner
• FakeSyncManager
• FakeTypedCursor (populates a TypedCursor with specified data)
• FakeEventBus
• FunctionalTestCase -- base class for functional tests, providing some helpful functionality:
• Greatly increases timeouts for UI and data operations to eliminate some sources of flakiness
• Optionally waits on the user list to load (controlled by mWaitForUserSync, default: true)
• Closes all activities when the test ends
• Provides a means to determine the current activity (getCurrentActivity())
• Provides a way to take a screenshot during the test if the tests are being run with Spoon (screenshot(String tag))
• Provides a means to wait for ProgressFragments, a Fragment which tracks and displays loading state, to be in the LOADED state before continuing (waitForProgressFragment())
• Provides a means to wait for the initial sync process, which occurs when the client database is unpopulated (waitForInitialSync())
• Provides means to "sleep", which otherwise doesn't necessarily work as expected during Espresso tests (checkViewDisplayedSoon(), checkViewDisplayedWithin(int ms))
• Idling resources (classes which can cause Espresso to delay its assertions until some condition is met)
• WifiStateIdlingResource - busy until Wifi is turned on
• EventBusIdlingResource - busy until a particular EventBus event fires
• ProgressFragmentIdlingResource - busy until a ProgressFragment is in the LOADED state
• SyncTestCase -- a FunctionalTestCase that clears user data at the beginning of every test and provides functionality for modifying Wifi state
• Matchers -- custom matchers for selecting Objects in ViewAssertions

Unit Tests

For new classes, every nontrivial, non-view (Activity, Fragment) class should be checked in with a set of unit tests covering all nontrivial public functions. This helps ensure the system's stability moving forward and maintains a baseline of test coverage.

At time of writing, most existing classes do not have unit tests, but there is some coverage, particularly for controller classes, which are the most user-facing of the non-view classes. The remaining gaps should be filled in over time.

Functional Tests

For each activity, major user-facing flows should be granted at least one functional test. These tests are far more time-consuming to run and can be less reliable than unit tests, however, so developers are encouraged to use functional tests primarily in cases where coverage cannot be achieved via unit tests.

Currently, there are some small gaps in the functional tests, but coverage is significantly better than unit test coverage. Tests exist for virtually every major flow, with some exceptions where limitations of Espresso or flakiness have prevented proper test operation. In these cases, TODO's have been left in the code to be addressed later on.

Common Pitfalls

There are some common pitfalls to be aware of when writing new tests for the client app or modifying existing tests.

IdlingResource persistence

In Espresso 1.1, IdlingResources persist until the end of a test suite and cannot be unregistered. Furthermore, IdlingResources with the same name are considered to be the same, and any duplicate IdlingResources are ignored. This can cause problems for IdlingResources based on one-time events (such as EventBusIdlingResource), as the IdlingResource remains even after the event fires. This is currently being mitigated by giving each EventBusIdlingResource a unique name, but it means that there could be dozens of idling resources remaining at the end of a test suite.

The long-term solution to this problem will be upgrading to Espresso 2.0, which allows for IdlingResource unregistration.

Timeouts

Some app operations, particularly initial sync, can take a very long time given enough data. Even with the current timeouts in FunctionalTestCase, which are set to 5 minutes, it's possible for an operation to timeout, causing the test to fail. The solution to this problem should ultimately be to fix the performance issue and not the tests, but for now, there is a chance of test flakiness given a sufficient amount of data.

UI pop-in

There are many cases within the app that elements change their displayed state after being loaded to match their expected value--for example, when a patient chart is opened, the patient chart appears before any observations, which actually includes the admission and symptoms onset dates. Unless Espresso is instructed to wait for these events, it will fail if the test asserts that these are displayed. This is commonly mitigated in the tests by using the FunctionalTestCase.checkViewDisplayedSoon() and/or FunctionalTestCase.checkViewDisplayedWithin(int ms) methods, which repeatedly check if a view is displayed until a specified timeout.

Test flakiness

In the past, many of the existing client tests were flaky. Through a concerted effort, this flakiness has been more or less eliminated, but it's possible that some tests may be flaky in rare occasions. Any common flakes or consistent failures likely point to a real issue, however.