[$250] Support compressed push notification payloads

[arosiclair]

• Parent issue: https://github.com/Expensify/Expensify/issues/286643

Background

We include Onyx data in the payload for each of our push notifications. On receipt, we merge the onyx data and update the app with any previously missed onyx updates if necessary (src). However, there is a 4 KB size limit for push notification payloads, so we remove the Onyx data when see the payload would be too large. This usually happens because the message for the push notification is too long. Additionally, we're implementing some optimizations in the backend where we squash multiple Onyx updates together to reduce the number updates we store in the database which also increases the potential payload size.

Problem

The 4KB size limit prevents Onyx data delivery in push notifications for large messages and prevents us from squashing onyx updates in some scenarios.

Solution

Support push notification payloads that are compressed using GZIP so that we can fit more data. Specifically, the value in payload would be a GZIP compressed JSON string. Decompressing and parsing the string as an object should yield the same payload object that we normally receive now.

To ensure backward compatibility we should still support uncompressed payloads. We'll start sending compressed push notification payloads when we're confident clients have broad support for it.

We'll need to support compressed payloads in both JS and native code:

• React Native
• NewDot Android
• NewDot iOS
• HybridApp Android
• HybridApp iOS

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~021876663566682945155
• Upwork Job ID: 1876663566682945155
• Last Price Increase: 2025-01-07
• Automatic offers:
• dominictb | Reviewer | 105934695

Issue Owner

Current Issue Owner: @adhorodyski

[melvin-bot]

Triggered auto assignment to @jliexpensify (NewFeature), see https://stackoverflowteams.com/c/expensify/questions/14418#:~:text=BugZero%20process%20steps%20for%20feature%20requests for more details. Please add this Feature request to a GH project, as outlined in the SO.

[adhorodyski]

Adam from Callstack, I'd like to work on this.

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~021876663566682945155

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @dominictb (External)

[dominictb]

Hi @adhorodyski, how's the progress so far?

[adhorodyski]

Hey, sorry for missing an update in here! I took the time to look around for solutions available, here's what I have so far:

Steps required for processing on each platform:

• convert the gzipped json string into the actual byte array
• decompress the gzip data
• convert decompressed data into a string
• parse json string into an object

Now the gzip decompression step is not so straightforward in Hermes (JS) because we're missing native utilities to do so. CompressionStream is only discussed so far as a non-ECMAScript feature to be added.
What I went with for now is a widely used utility library pako that will likely need to go through a regular review, I couldn't find any approach for this problem within E/App.
Another approach we might consider would be to expose this functionality from the native layer.

I also started with the JS implementation working on this branch: https://github.com/callstack-internal/Expensify-App/tree/feat/compressed-push-notifications (here's the current diff)

[adhorodyski]

@arosiclair actually before I proceed further, I'd like to ask one crucial question:

• Have we considered a case where even the compressed payload exceeds the 4kb limit? I think we should prepare for this as being inevitable sooner or later as we can't easily predict how big updates are going to be sent down to the client's app. Plus, notifications are not guaranteed to be delivered.

Is it possible that we take an alternative approach and only send down what the client needs to fetch itself from the API? Have this been considered before?

[arosiclair]

Have we considered a case where even the compressed payload exceeds the 4kb limit? I think we should prepare for this as being inevitable sooner or later as we can't easily predict how big updates are going to be sent down to the client's app.

Yeah we handle on this on the backend. You can expect payload data in every push notification. The onyxData may or may not be present in the payload, but that case is already handled so we shouldn't have to worry about that for this.

Is it possible that we take an alternative approach and only send down what the client needs to fetch itself from the API?

We are doing this already actually. The payload will contain some onyx updates and the client will attempt to merge them. If it detects that there are other missed onyx updates, those will be fetched from the API using GetMissingOnyxMessages. So we're attempting to push as much data as we can and any other updates will be fetched where necessary.

[adhorodyski]

Oh sorry if I phrased this poorly - I was thinking "can we send as little as possible in the notification" so that we don't actually have to try walk around the limit? Offloading this from the notification to the device can not only simplify it, but also save us the trouble in the future. I'm worried we'll eventually hit the wall even with gzipped content and will be back at square 1.
I can imagine we only send an ID/IDs of what the client needs to fetch itself, without even having to deal with JSON structures in the payload. Am I missing some important piece here?

[arosiclair]

I see what you mean. For context, our philosophy has been to use push notifications as another channel for Onyx data delivery. Push notifications are special since they can push updates to the app even when it isn't running and even when the connection is slow/unreliable. So we try to leverage that to keep the client as up to date as possible.

For now, we shouldn't worry too much about further payload size increases down the line. We have lots of integration tests to make sure that doesn't happen (at least not by accident). If it does become an issue again, we'll just start another discussion on how to move forward. This issue was born from one of those discussions for example.

[adhorodyski]

Oki, that makes sense now! Just a heads up we might be able to leverage https://github.com/Expensify/App/tree/main/modules/background-task just for that if we decide it's not enough, cc @szymonrybczak who authored it. It was designed to use best available APIs on each platform to handle background jobs, but I know iOS has some design limitations when we close the app completely.

I now know fully what we're dealing with, thanks @arosiclair. I will continue with Android now on the linked branch and add unit tests on the JS for the parsing.

[adhorodyski]

Update: I pushed Android's implementation today, will test this tomorrow together with web.

[adhorodyski]

I was able to verify today that Android passes fine, working on the iOS implementation now. Discussed with @staszekscp that for Hybrid we'd need to:

• copy the implementation for Android
• directly reference the ND code on iOS

[adhorodyski]

Update: I think I've completed the iOS version, will see how we can go about testing this one as well as plug it in into Hybrid.
Current diff here.

[adhorodyski]

Hey! As I'll be off next week for the onshore event @VickyStash will take over this issue to finish up the Hybrid integration.

[jliexpensify]

Feel free to comment @VickyStash and I'll assign you

[VickyStash]

Hi, I'm Viktoryia from Callstack - expert contributor group - and I would like to work on this issue.

[melvin-bot]

📣 @dominictb 🎉 An offer has been automatically sent to your Upwork account for the Reviewer role 🎉 Thanks for contributing to the Expensify app!

Offer link
Upwork job

[jliexpensify]

Assigned you @VickyStash

[VickyStash]

Updates:

• tested ios standalone app over push notifications with compressed/not compressed payload - works as expected.

  videos

  ios_standalone_gzip_format.mp4

  ios_standalone_old_format.mp4

• added minor improvements to Android implementation:

  • use util.Base64 from android, not java, to get rid of SDK version limitations
  • increased buffer size to 4kb

• updated unit test to use more realistic data

• opened Draft PR for the E/App repo

• Implemented Android updates for the HybridApp (diff)

TODOs:

• Implement iOS updates for the HybridApp.
• Prepare PRs for both E/App and HybridApp repos.

[VickyStash]

Updates:
Today I was focused on iOS HybridApp implementation. Direct reference of the ND code didn't work right away as the app crashed silently any time I tried to get the push notification with compressed payload, so I've spent some time on investigations.
The reason turned out to be in the app/managers/PushNotificationManager.js, where the app wrongly checked for the payload.type and passed the notification to be processed as local one on the native side.
I need to add the notification payload decompressing to that file so all of the checks work correctly.

If there are no more unexpected crashes/issues - the PRs should be ready for review on Tue/Wed.

[arosiclair]

The reason turned out to be in the app/managers/PushNotificationManager.js, where the app wrongly checked for the payload.type and passed the notification to be processed as local one on the native side.

Oh yeah FYI, we're working on fixing that here

[VickyStash]

Updates

• updated PushNotificationManager in the Hybrid app to support compressed payload
• added updates to the Hybrid app iOS. NOTE: I haven't replaced the current NotificationService.swift implementation with direct reference to the ND file, cause to do that we need to add an additional method to extract appGroup param from .entitlements file. If needed we can do it as a follow up, but if you think it's better to do it now just let me know @arosiclair
• opened Hybrid App Draft PR and started preparing it to the review
• added test steps to the E/App PR, just need to add recordings

TODOs:
just need to prepare screenshots/recordings for both PRs

[arosiclair]

I haven't replaced the current NotificationService.swift implementation with direct reference to the ND file, cause to do that we need to add an additional method to extract appGroup param from .entitlements file. If needed we can do it as a follow up, but if you think it's better to do it now just let me know @arosiclair

That's fine no need for that now 👍

[VickyStash]

Updates:
Both PRs have been opened for the review.
E/App: #56154
Mobile-Expensify: https://github.com/Expensify/Mobile-Expensify/pull/13406