[FR] [Implementation] Smart Limits for Detection Rules

[shashank-elastic]

Repository Feature

Core Repo - (rule management, validation, testing, lib, cicd, etc.)

Problem Description

Per discussion(s) in - #4150, the previous limit for rule assets was removed to introduce historical package versions in 8.17.1 and 8.17.2. With this change the current asset count in rule package is close ~11500 and the tested asset limit on serverless projects is ~15000 before we hit OOM issues on serverless instances.

We need to reintroduce "smart limits" on detection rules to ensure there is minimum disruption to rule version diff feature in product that is used to analyse rule upgrades.

Desired Solution

With some brainstorming ides on the SWG call, there are 2 possible solutions to this issue.

Solution 1

• Reintroduce limits to hold have at most 5 versions of each rule
• More than 5 versions, start removing the oldest version of the rule.

Solution 2

• When the total assets increase 15000 rule assets, removing the excess assets and bring it back to the desired tested number
• When package is built if we have 15010 rules, then the excess 10 rule assets which are of the oldest nature will be removed
  • With this we could potentially remove more than 1 version of the same rule becuase its the oldest in nature
  • We should ensure latest versions of the rule even if its oldest is retained.

For both of these solutions we need to revist code at keep_latest_versions

Considered Alternatives

Be able to handle rule asset management from the product side, this is being considered as a solution to ensure there is enough time for product enhancements

Additional Context

#4150