Client Sync

Note: Sync is being overhauled. See Fixing Sync for Project Buendia

Summary

The Buendia client application currently retrieves virtually all of its remote data by performing sync operations with the Buendia server. Sync may be triggered in a number of different ways, with a variety user-facing effects. This document describes these various sync processes in detail.

Sync Account Creation

Sync operations rely on a sync account to be registered. Whenever a PatientListFragment is attached to a PatientSearchActivity (the Locations activity, Round activity, and "All Present Patients" activity), the PatientListFragment makes a request to GenericAccountService to check for a Buendia sync account and create one if one isn't present. If the account is newly created or the account did not set a SETUP_COMPLETE preference, then this will also trigger an initial sync. For more information, see Initial Sync.

Note that there is no longer a good reason why account creation is part of PatientListFragment; this is the result of legacy code but has been a low-priority fix to make. For more information, see the section in Known Issues.

Data Model Availability

Most of the app assumes that data from the server is already available locally in the client database. When this assertion needs to be checked (as in the Locations controller, TentSelectionController), we call AppModel.isDataModelAvailable(), which makes a determination of whether data has been downloaded from the server. This data need not be up-to-date, but must be present in some form.

When all data is downloaded from the server (a "full" or "initial" sync), we record SYNC_START_TIME and SYNC_END_TIME timestamps to a Misc table in the database, which are written at the very beginning and very end of the initial sync process. If both timestamps are present and SYNC_END_TIME > SYNC_START_TIME, then we make the assumption that all data in between has been fetched successfully, and therefore the data model is available.

Initial Sync

When the app performs an initial sync, it makes RPC's to the Buendia server to retrieve all of the following data:

• Concepts
• Patients
• Chart definitions
• Users
• Locations
• Patient encounters

Triggering

Initial sync is triggered in the following scenarios:

• When the account is first created (see Sync Account Creation)
• When a patient list is opened, the account has been created, but account setup has not completed successfully
• When the Locations activity (TentSelectionActivity) is loaded, the app model is considered unavailable (see Data Model Availability), and no other sync is in progress
• When the Locations controller (TentSelectionController) detects that a sync has completed, but the data model is still considered unavailable
• When the Locations controller detects that a sync has completed, and the data model is considered available, but no locations are present

UI Effects

Initial sync is considered required for all user actions except for user login, so the user is blocked from accessing any activities other than Settings or User Login until initial sync has completed and the data model is available (see Data Model Availability).

While initial sync is in progress, the SyncAdapter will periodically send progress updates to the BroadcastReceiver in SyncManager. When these progress updates occur, they are forwarded to the BaseActivity, which shows sync progress with a progress bar. Note that the user may be shown a spinner for a short time until a sync progress message is received.

If initial sync fails, the user is shown a dialog providing options to return to user selection, change settings, or retry the sync.

The user may also choose to cancel an initial sync by pressing the close button in the top-right corner. Cancelling the sync is asynchronous process and is not handled immediately by the sync adapter; rather, the SyncManager sends the SyncAdapter a request to cancel, and the SyncAdapter then performs cancellation at the soonest opportunity, sending a message back to the SyncManager that cancellation was successful. Currently, the app awaits this message before actually returning to the user screen.

Periodic Sync

Once the account is created (see Sync Account Creation), a periodic sync will occur every 5 min. Periodic syncs request the same information as initial sync, with one caveat: observations are updated incrementally, rather than syncing all observations. For more information, see Incremental Observation Sync.

In theory, periodic sync should have no user-facing effect until it is completed, at which point the new data would become available (e.g. patient list updating its patients). Unfortunately, however, periodic sync can have a significant user-facing impact--since the SyncAdapter is not thread-safe and designed to be run serially, periodic syncs actually block any other requested syncs. This can have a measurable effect on performance in the chart view and when requesting syncs in the patient list. For more information, see the entry in the Known Issues section.

Observation Sync

When viewing a patient chart, observations are requested from the server every minute. Whenever any new observations are fetched, the patient chart reloads to display them. Note that "new" observations may be historical data, but entered since the last time a sync was performed.

An observation sync is currently also requested after a patient is created, which submits some initial observations alongside the patient.

Incremental Observation Sync

To speed up the sync process, the server endpoint for fetching observations accepts an 'sm' parameter denoting the time of the latest patient encounter the client is aware of. The server then sends only the observations newer than 'sm'. The value used for 'sm' is recorded by the SyncAdapter after each observation sync and stored in the client database.

"Pulldown" Sync

The user may manually request a sync by swiping down in any patient list (via a SwipeRefreshLayout). Doing so is equivalent to a periodic sync, with a couple of minor exceptions:

• If the sync times out or fails, the user will be notified via a "BigToast" message (our custom Toast replacement)
• A refresh indicator is shown while the sync is in progress

Known Issues

Sync account is registered in PatientListFragment

Currently, the sync account is created/registered in PatientListFragment for the legacy reason that it used to be part of the first activity. There is no longer a compelling reason for this logic to be in PatientListFragment, and it could just as easily be moved to another part of the app, but this refactoring has been considered low priority relative to its risk.

Sync transactions unreliable

SQLite savepoints are being used to make the entire sync process transactional, and this was the original model that was meant to be used to determine data model availability; however, we have observed situations in which the savepoints are not working as intended--data can remain written even when the savepoint is released. This problem has been mitigated, however. For more information, see the descriptions of SYNC_START_TIME and SYNC_END_TIME in Data Model Availability.

User actions may be blocked by periodic or "pulldown" syncs

If any sync is in progress, then any subsequent sync requests will be blocked until that sync completes. Normally, this doesn't matter much, but there are some cases where it can create problems.

For example, when a patient is created, we currently don't immediately store the initial observations in the database, but instead rely on a sync to retrieve them from the server. If there is a periodic or "pulldown" sync occurring, then these observations may not be fetched from the server for seconds or even minutes, causing some information to be missing from the patient chart (namely admission date and symptoms onset date) until the existing sync is completed.

Another example is a "pulldown" sync requested while a periodic sync is in progress--technically, the requested sync won't occur until the periodic sync is in progress; however, in this case, the refresh indicator will disappear after the first sync, so the user impact is negligible.

Cancelling a sync does not cancel network requests

Currently, cancelling a sync does not also cancel Volley requests, so sync may hang for an extended period of time waiting for Volley requests to timeout before completing cancellation.

Initial sync does not scale well

Initial sync scales linearly with the number of patients, observations, etc. on the server. For small numbers of observations, this is fine, but for a server with thousands of observations (or more), initial sync can start to take several minutes. Network requests from the client eventually time out, so there is a certain point where server requests may be slow enough that client sync cannot complete (during testing, this threshold was about 20,000 observations, but this depends on a lot of variables, such as network bandwidth and what hardware the server is running on).

It should be possible to mitigate this issue by changing the initial sync process to only request some subset of observations (such as the latest for each patient), only downloading the rest of the observations when needed. This will require more thought as well as changes on the server to support this use case.