Import from Donation Backup Service #1253

aminlatifi · 2024-01-18T16:59:50Z

…sion Donation save backup import revision

aminlatifi · 2024-01-22T11:40:21Z

src/services/cronJobs/backupDonationImportJob.ts

+        logger.error(`donation error with id ${donation._id}: `, e);
+        logger.error('donation error with params: ', donation);


There is a point here, if importing an item gets an error, it will be fetched in the next round of importing again. So, it will endlessly be processed in each iteration.

@aminlatifi
How about adding another field(like isImported) for this kind of donations and set it true in this case, and not get them again

The name can be importError I think to hold the error message as well. Before going for that, we must be sure this importing is a deterministic process. I mean, if it fails here, it will fail every next time we try. If it holds, that would be ok to mark the items got error in importing and ignore them, but if not we must allow them to be processed in next iteration.

Cool , I will do it

I see the skip was removed. Could simply add a counter that increases when it enters the catch() statement. And on each iteration reset it to 0. And skip based on that number.

But at the same time we could simply increase the limit and let the process run every hour or so. It shouldn't be a heavy process.
I was checking failed donations, and even on the previous round where we had big errors, the amount is never a huge number.

aminlatifi · 2024-01-22T11:45:06Z

src/services/cronJobs/backupDonationImportJob.ts

+    skip += limit;
+    donations =
+      await getDonationSaveBackupAdapter().getNotImportedDonationsFromBackup({
+        limit,
+        skip,
+      });


There is a serious issue here, the items we import above, will not match the query filter any more (their imported field will have true value). So, we shouldn't increase the skip value here naively, just increase by number of items which we got error in processing above.

Reuse donation resolver

…kup-import

CarlosQ96

Thanks Mohammad, those are the changes I was wanted. Better error handling, interfaces and mocks.
I think we should probably merge, check there isn't any issue in staging.

* Verify transfer spl-token on solana, supporting multi chain solana related to #1175 * fix solana tests on devnet * add solana additional tokens * fix solana token transfer test * improve joi schema chaintype validator * Add another spl-token transfer test case, change validator of createDonation webservice related to #1232 (comment) * Refactor and change a nested if-else with witch case * Removed only tags from tests * Update src/utils/errorMessages.ts Co-authored-by: Amin Latifi <[email protected]> * Update src/utils/locales/en.json Co-authored-by: Amin Latifi <[email protected]> * Fill networkId for solana addresses when create/update projects * Add some test tokens for solana dev chain * Import lost donations into DB cronjob (#1236) * import donations from env vars and add tests * Read LOST_DONATIONS_TX_HASHES from .env instead of config * Fix build error --------- Co-authored-by: Amin Latifi <[email protected]> Co-authored-by: Mohammad Ranjbar Z <[email protected]> * setup notification center disabling and reduce notifications * Change filling networkId for solana * Fix migration files for adding spl tokens on solana * Modify create donation test cases for check filling networkId of solana donations * comment out notification center methods interface * Revert "comment out notification center methods interface" This reverts commit 4a6dd8c. * set logic as before, comment notification center methods not required * move donation received logic to notification adapter * Refactor get networkId for solana addresses and implement getAppropriateNetworkId * Fix filling solana donations price (#1252) * Fix filling solana donations price related to Giveth/giveth-dapps-v2#3394 (comment) * Add informative logs for filling value usd part * Add log for createDonation webservice * change type of transactionNetworkId to number in ajv schema * Update createDonation to use correct networkId * Added jobId to donation verification queue * fix test env * Import from Donation Backup Service (#1253) * Added Donation Save Backup Adapter * add backup service import cronjob * add interface of used params from mongo data * Change loading urls for backup service * Refactor donationSaveBackupAdapter and adding mock adapter * Removed unused package script * Refactored createBackupDonation to reuse resolver * Fix createBackupDonation() and write test case for that * Add importError to failed donation mongo backup --------- Co-authored-by: mohammadranjbarz <[email protected]> Co-authored-by: Carlos <[email protected]> * Fix query of getting failed donations from mongo API * Add more logs for importing failed donations * Change info logs to debug logs * 1.21.0 * Add importDate to donation entity * Fill importDate of donation correctly --------- Co-authored-by: Mohammad Ranjbar Z <[email protected]> Co-authored-by: Carlos <[email protected]> Co-authored-by: CarlosQ96 <[email protected]>

* Verify transfer spl-token on solana, supporting multi chain solana related to #1175 * fix solana tests on devnet * add solana additional tokens * fix solana token transfer test * improve joi schema chaintype validator * Add another spl-token transfer test case, change validator of createDonation webservice related to #1232 (comment) * Refactor and change a nested if-else with witch case * Removed only tags from tests * Update src/utils/errorMessages.ts Co-authored-by: Amin Latifi <[email protected]> * Update src/utils/locales/en.json Co-authored-by: Amin Latifi <[email protected]> * Fill networkId for solana addresses when create/update projects * Add some test tokens for solana dev chain * Import lost donations into DB cronjob (#1236) * import donations from env vars and add tests * Read LOST_DONATIONS_TX_HASHES from .env instead of config * Fix build error --------- Co-authored-by: Amin Latifi <[email protected]> Co-authored-by: Mohammad Ranjbar Z <[email protected]> * setup notification center disabling and reduce notifications * Change filling networkId for solana * Fix migration files for adding spl tokens on solana * Modify create donation test cases for check filling networkId of solana donations * comment out notification center methods interface * Revert "comment out notification center methods interface" This reverts commit 4a6dd8c. * set logic as before, comment notification center methods not required * move donation received logic to notification adapter * Refactor get networkId for solana addresses and implement getAppropriateNetworkId * Fix filling solana donations price (#1252) * Fix filling solana donations price related to Giveth/giveth-dapps-v2#3394 (comment) * Add informative logs for filling value usd part * Add log for createDonation webservice * change type of transactionNetworkId to number in ajv schema * Update createDonation to use correct networkId * Added jobId to donation verification queue * fix test env * Import from Donation Backup Service (#1253) * Added Donation Save Backup Adapter * add backup service import cronjob * add interface of used params from mongo data * Change loading urls for backup service * Refactor donationSaveBackupAdapter and adding mock adapter * Removed unused package script * Refactored createBackupDonation to reuse resolver * Fix createBackupDonation() and write test case for that * Add importError to failed donation mongo backup --------- Co-authored-by: mohammadranjbarz <[email protected]> Co-authored-by: Carlos <[email protected]> * Fix query of getting failed donations from mongo API * Add more logs for importing failed donations * Change info logs to debug logs * 1.21.0 * Add importDate to donation entity * Fill importDate of donation correctly * Put importing failed donations in try...catch * Added back jobId to verifyDonationsQueue, along with removeOnComplete and removeOnFail --------- Co-authored-by: Mohammad Ranjbar Z <[email protected]> Co-authored-by: Carlos <[email protected]> Co-authored-by: CarlosQ96 <[email protected]>

Added Donation Save Backup Adapter

40a51df

aminlatifi marked this pull request as draft January 18, 2024 17:00

CarlosQ96 force-pushed the donation-save-backup-import branch 7 times, most recently from f103a88 to 05b63ff Compare January 21, 2024 05:35

add backup service import cronjob

afca023

CarlosQ96 force-pushed the donation-save-backup-import branch from 05b63ff to afca023 Compare January 21, 2024 05:45

CarlosQ96 and others added 4 commits January 21, 2024 01:02

add interface of used params from mongo data

af6cda5

Change loading urls for backup service

cad6c78

Refactor donationSaveBackupAdapter and adding mock adapter

1b33df3

Merge pull request #1255 from Giveth/donation-save-backup-import_revi…

edcf0bb

…sion Donation save backup import revision

aminlatifi marked this pull request as ready for review January 22, 2024 07:33

aminlatifi added 2 commits January 22, 2024 11:05

Removed unused package script

7f565a1

Refactored createBackupDonation to reuse resolver

487ba84

aminlatifi marked this pull request as draft January 22, 2024 08:57

Fix createBackupDonation() and write test case for that

7fe5d30

aminlatifi mentioned this pull request Jan 22, 2024

Reuse donation resolver #1257

Merged

aminlatifi commented Jan 22, 2024

View reviewed changes

aminlatifi and others added 5 commits January 22, 2024 15:17

Merge pull request #1257 from Giveth/refactor-create-backup-donation

6831781

Reuse donation resolver

Add importError to failed donation mongo backup

dfbea04

Merge branch 'refactor-create-backup-donation' into donation-save-bac…

8f18d08

…kup-import

fix merge conflicts with staging

6f12dff

fix test env

bc5b4cc

CarlosQ96 marked this pull request as ready for review January 23, 2024 05:02

CarlosQ96 approved these changes Jan 23, 2024

View reviewed changes

mohammadranjbarz merged commit de14c4b into staging Jan 23, 2024
3 checks passed

mohammadranjbarz deleted the donation-save-backup-import branch January 23, 2024 08:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Import from Donation Backup Service #1253

Import from Donation Backup Service #1253

aminlatifi commented Jan 18, 2024 •

edited

Loading

aminlatifi Jan 22, 2024

mohammadranjbarz Jan 22, 2024

aminlatifi Jan 22, 2024

mohammadranjbarz Jan 22, 2024

CarlosQ96 Jan 23, 2024 •

edited

Loading

aminlatifi Jan 22, 2024

mohammadranjbarz Jan 22, 2024

CarlosQ96 left a comment •

edited

Loading

		logger.error(`donation error with id ${donation._id}: `, e);
		logger.error('donation error with params: ', donation);

Import from Donation Backup Service #1253

Import from Donation Backup Service #1253

Conversation

aminlatifi commented Jan 18, 2024 • edited Loading

aminlatifi Jan 22, 2024

Choose a reason for hiding this comment

mohammadranjbarz Jan 22, 2024

Choose a reason for hiding this comment

aminlatifi Jan 22, 2024

Choose a reason for hiding this comment

mohammadranjbarz Jan 22, 2024

Choose a reason for hiding this comment

CarlosQ96 Jan 23, 2024 • edited Loading

Choose a reason for hiding this comment

aminlatifi Jan 22, 2024

Choose a reason for hiding this comment

mohammadranjbarz Jan 22, 2024

Choose a reason for hiding this comment

CarlosQ96 left a comment • edited Loading

Choose a reason for hiding this comment

aminlatifi commented Jan 18, 2024 •

edited

Loading

CarlosQ96 Jan 23, 2024 •

edited

Loading

CarlosQ96 left a comment •

edited

Loading