Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
FEAT: Decrease run-time duration of ODS-QLIK Process (#28)
This change alters the way ods-qlik data is loaded into the dmap-import database. The new process does the following: - download batches of cdc csv.gz files on to local machine - merge file batches into a single csv file - load csv file to _history table - convert csv file to dataframe object - perform INSERT - UPDATE - DELETE operations on _fact table individually with dataframe object The new process completely removes the need for a complicated query to load data from _history table to the _fact table and thus dramatically reduces loading time when running the ods-qlik process.
- Loading branch information