Skip to content

Commit

Permalink
FEAT: Decrease run-time duration of ODS-QLIK Process (#28)
Browse files Browse the repository at this point in the history
This change alters the way ods-qlik data is loaded into the dmap-import database.

The new process does the following:
- download batches of cdc csv.gz files on to local machine
- merge file batches into a single csv file
- load csv file to _history table
- convert csv file to dataframe object
- perform INSERT - UPDATE - DELETE operations on _fact table individually with dataframe object

The new process completely removes the need for a complicated query to load data from _history table to the _fact table and thus dramatically reduces loading time when running the ods-qlik process.
  • Loading branch information
rymarczy authored Nov 7, 2024
1 parent 5e01822 commit 9f9ab45
Show file tree
Hide file tree
Showing 4 changed files with 552 additions and 261 deletions.
Loading

0 comments on commit 9f9ab45

Please sign in to comment.