[TDL-22683] Use Date Windowing Pagination #113

shantanu73 · 2024-02-20T10:12:25Z

Description of change

Implemented date windowing for all multi-threaded streams to optimize the process of fetching stream data and resolve infinite loop issues in extraction runs.
Added a limit to buffer size of fetched records to release so that memory consumption can be optimized.
Removed performance metrics calculation which was adding additional overhead on tap performance.
Fixed data loss issue in interrupted sync scenario.

Manual QA steps

Ran sync runs, and python memory profiling to determine tap memory consumption and stream performance.

Risks

Rollback steps

revert this branch

1) Fixed Date windowing logic for multithreaded_bookmark_generator. 2) Added date_windowing instance variable as true in the constructor of activities stream.

1) Removed endpoint params from activities generator. 2) Overrode modify_reques_params method for Loan Transactions generator. 3) Removed the method _all_fetch_batch_steps from multithreaded bookmark generator. 4) Modified the implementation for _all_fetch_batch_steps method in multithreaded offset generator to include date windowing. 5) Added new method modify_reques_params in multithreaded offset generator.

1) Set max threads back to 20. 2) Fixed bug in Date windowing implementation.

1) Changed number of threads from 5 to default 20 for gl_journal_entries stream. 2) Added date window implementation for gl_journal_entries stream.

…ing streams.

… Added 1 second in the BEFORE value.

…eams.

… stream class.

…tream class.

1) Removed redundant initialization of date_windowing for Activities, GL Journal entries & Loan transactions streams. 2) Changed max threads for Deposit transactions stream to default value.

- Remove redundant code

- Minor refactoring

- Revert previous integration test changes

rdeshmukh15 · 2024-05-08T07:21:43Z

tap_mambu/helpers/client.py

        self.__username = username
        self.__password = password
        self.__subdomain = subdomain
        base_url = "https://{}.mambu.com/api".format(subdomain)
        self.base_url = base_url
        self.page_size = page_size
+        self.window_size=window_size


Maintain the spacing as on previous lines

Addressed already.

rdeshmukh15 · 2024-05-08T07:30:01Z

tests/test_sync_canary.py

@@ -1,35 +0,0 @@
-"""


Any reason to remove these tests completely?

Removed it because it was increasing execution time and not adding much value to the test suite.

rdeshmukh15 · 2024-05-08T07:32:32Z

tests/test_pagination.py

@@ -16,6 +16,7 @@ def name():

    def untestable_streams(self):
        return set([
+            "clients", # Stream does not have enough records to test pagination


Are there only a few records, or do we have no records at all? If we do have a small number of records, we can reduce the page limit and test this specific stream.

window_size implementation fixed the issue, removing clients from untestable_streams().

rdeshmukh15 · 2024-05-08T08:15:52Z

tap_mambu/helpers/client.py

        self.__username = username
        self.__password = password
        self.__subdomain = subdomain
        base_url = "https://{}.mambu.com/api".format(subdomain)
        self.base_url = base_url
        self.page_size = page_size
+        try:
+            self.window_size = int(float(window_size)) if window_size else DEFAULT_DATE_WINDOW_SIZE


Can't we do integer typecasting directly instead of, converting to float first?

>>> int("10.0") Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: invalid literal for int() with base 10: '10.0'

rdeshmukh15 · 2024-05-08T08:38:39Z

tap_mambu/tap_generators/generator.py

@@ -1,10 +1,11 @@
+import time


Remove the unused import time

cosimon and others added 30 commits December 20, 2023 18:14

Adding None support to HashableDict

d51282c

Initial date windowing logic for activities stream.

7801007

Changes:

e3851a8

1) Fixed Date windowing logic for multithreaded_bookmark_generator. 2) Added date_windowing instance variable as true in the constructor of activities stream.

Changing number of threads from 20 to 1 to fix memory issue.

859d923

Changes:

db922e5

1) Set max threads back to 20. 2) Fixed bug in Date windowing implementation.

Changes:

01d315d

1) Changed number of threads from 5 to default 20 for gl_journal_entries stream. 2) Added date window implementation for gl_journal_entries stream.

Fixed issue with initializing offset and infinite loop in date window…

28fbc3a

…ing streams.

Remove performance metrics

e33022b

Remove performance metrics

887e0b3

Initializing final_buffer variable.

1b399dc

Fixed BEFORE date for loan_transactions & gl_journal_entries streams.

da24d92

Fixed BEFORE date for loan_transactions & gl_journal_entries streams.…

a8d46dd

… Added 1 second in the BEFORE value.

Limit max buffer size

6d6ae66

Add generator change

162e542

Changing default date window, from 3 to 5.

b5c3941

Added generic date windowing for all MultiThreadedOffsetGenerator str…

58a98b8

…eams.

Implemented modify_request_params method for Activities stream.

3403ea7

Implemented modify_request_params method for Installments stream.

aedd6aa

Disabled Date windowing for Users stream.

a9ff3ad

Removed endpoint filter criteria for Clients stream.

db07f7a

Implemented modify_request_params method for Communications stream.

554720b

Removed endpoint filter criteria for DepositAccounts stream.

7711832

Removed endpoint filter criteria for DepositTransactions stream.

11e44de

Removed modify_request_params implementation from GL Journal Accounts…

09473ff

… stream class.

Removed endpoint filter criteria for Groups stream.

232a7ea

Removed endpoint filter criteria for Interest Accrual Breakdown stream.

70fc616

Removed modify_request_params implementation from Loan Transactions s…

7856961

…tream class.

Changes:

b4cf6d5

1) Removed redundant initialization of date_windowing for Activities, GL Journal entries & Loan transactions streams. 2) Changed max threads for Deposit transactions stream to default value.

Fixed bug in communications stream date windowing implementation.

69f2435

RushiT0122 and others added 7 commits April 12, 2024 20:49

Minor fixes

b1e9138

Remove wait for thread implementation

0efeb77

- Empty buffer before next date window

ba83d27

- Remove redundant code

Remove frequent bookmarking

179ad28

Reduce integration test execution time

0aabdaf

Fix integration tests

77aa759

Merge cobnflicts

8cb2a43

RushiT0122 force-pushed the TDL-22683-date-windowing-pagination branch from bbeec12 to c4232d6 Compare May 3, 2024 13:19

Parallelise integration tests

5b691a9

RushiT0122 force-pushed the TDL-22683-date-windowing-pagination branch from c4232d6 to 5b691a9 Compare May 3, 2024 13:21

RushiT0122 added 6 commits May 3, 2024 13:21

fix config.yml

cd3ae43

fix config.yml

a680071

Read window_ize from config

8eae933

Minor window size fixes

fa9d9e3

- Fix bookmarking for FULL_TABLE multi-threaded streams

c37fedf

- Minor refactoring

- Add 30 days window size to reduce integration test execution time

f101dff

- Revert previous integration test changes

rdeshmukh15 reviewed May 8, 2024

View reviewed changes

RushiT0122 added 4 commits May 8, 2024 08:02

Update window_size implementation

993883a

Add window_size unittests

8a25625

Update README file

4fe5582

Install parameterized madule for unittests

9483813

rdeshmukh15 reviewed May 8, 2024

View reviewed changes

Fix review comments

f82eac5

rdeshmukh15 approved these changes May 10, 2024

View reviewed changes

Bump version 4.2.0

da2f222

rdeshmukh15 approved these changes May 13, 2024

View reviewed changes

RushiT0122 merged commit 79a9963 into master May 13, 2024
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TDL-22683] Use Date Windowing Pagination #113

[TDL-22683] Use Date Windowing Pagination #113

shantanu73 commented Feb 20, 2024 •

edited by RushiT0122

Loading

rdeshmukh15 May 8, 2024

RushiT0122 May 8, 2024

rdeshmukh15 May 8, 2024

RushiT0122 May 8, 2024

rdeshmukh15 May 8, 2024

RushiT0122 May 8, 2024

rdeshmukh15 May 8, 2024

RushiT0122 May 8, 2024 •

edited

Loading

rdeshmukh15 May 8, 2024

[TDL-22683] Use Date Windowing Pagination #113

[TDL-22683] Use Date Windowing Pagination #113

Conversation

shantanu73 commented Feb 20, 2024 • edited by RushiT0122 Loading

Description of change

Manual QA steps

Risks

Rollback steps

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RushiT0122 May 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shantanu73 commented Feb 20, 2024 •

edited by RushiT0122

Loading

RushiT0122 May 8, 2024 •

edited

Loading