Lower Impact KBs have on Database #2020
Labels
Comments
originalname51
added a commit
that referenced
this issue
Mar 12, 2021
The KB from the database is not required to be saved to function. This will save a large data-call to the database as the rule blob is saved as a large text file. #2020
originalname51
added a commit
that referenced
this issue
Mar 12, 2021
originalname51
added a commit
that referenced
this issue
May 14, 2021
* Update the rule engine to never save the KB blob to the database. The KB from the database is not required to be saved to function. This will save a large data-call to the database as the rule blob is saved as a large text file. #2020 * Slight cleanup for #2020 * Update rule runner for modular process and read/write separation. ** Update knowledge bases to be broken into sets of 10,000 * Add a way to re-balance rules. When a rule or watchlist item is added the rules will re-balance. This will page through all rule and watchlist items and assign them to knowledge bases. ** Update rule runner to accept multiple KBs of watchlist or UDRs (instead of just one of each) ** Break rule runner scheduler into several stepped parts to separate out fact gathering, kb processes, hit detail generation, and hit detail persistence. * Separate out updated rule runner parts to only read from the database with no modifications. ** Update KB generation to not hook in individual rules to the KB, but instead use the UDR * Update rule runner to go horizontal. These code changes are to assist in horizontal drools rules. Instead of using vertical scaling threads the rule engine for drools switch over to a queue based system. Hits are also always persisted in an async manner. This allows the rule engine to process messages on a queue instead of a scheduled. Some infrastructure changes were also done in order to support the new rule runing application * Add the gtas-rule-runner to the GTAS project. Break out rule runner from the rule engine. This changes from a scheduled task to a queue based system. There are no write executions on the gtas-rule-runner which allow for scaling horizontally and vertically. * Docker updates: Update ETL job dockerfile reference (was older reference which was preventing builds Add local properties for rule runner Remove HTTP-PROXY as it has been superceded by the gtas-ui. * Update local deployment *to reference gtas-rule-runner instead of rule-runner. * Update ConditionalOnProperty for Notificaiton Services To use conditional on property to use email.hit.notification instead of the manual enable.email.notification", name = "service was incorrectly erroring when running the rules due to not instantiating the HitEmailNotificationService. * Upload loader jar to rule-runner * Upload gtas-information-share jar * Update gtas-commons * Update gtas-parsers * Move GTAS-Rule-Runner to gtas-parent Gtas-rule-runner was very tightly coupled with gtas-parent and so it made the most sense to add it as a module instead of a separate project. * update docker compose and local docker compose * Add gtas-information-share * Updates to Rule Runner Updated rule runner to return a list of hit details instead of raw results that would need to be converted to hit details. Updated the fuzzy matcher to return a list of hit details instead of persisting the list. This tightens the logic on what happens on the final step of the queue. Updated hit detail's hitmakerid to not be json ignored. This allows the hit maker id to be used by the rule engine. Updated notification service instantiation to be agnostic to whether the email service was toggled on or off. * Add scripts for db change. * Fix test * Updates for deployment Control memory control dev deployment control local deployment * Update main.yml * Minor adjustments and addendums to docker related stuff * memory Co-authored-by: simbam1 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
As a database administer I want to have smaller queries requested if possible.
Please check to see if saving the binary KB and rule drl to the database.
The text was updated successfully, but these errors were encountered: