[Investigation] CI Check for Minstacked Integration Schema Changes

[Mikaayenson]

Repository Feature

Core Repo - (rule management, validation, testing, lib, cicd, etc.)

Problem Description

Currently when we update the integration schemas for our rules and try to release new content, our automation workflows break if the integration introduces a new min stack. We then have to min stack our rules (e.g. #4156) but this is after we've tried to version lock our rules and noticed double bumps.

We can potentially catch these changes earlier.

Desired Solution

By looking at the response from https://epr.elastic.co/search?prerelease=true we can potentially alert when the conditions.kibana.version changes for a given integration name.

Ideally, per PR, we could:

1. Load the rules in the PR
2. Pull all the integrations used in the rules
3. Pull the conditions.kibana.version in the manifest that we have (assuming it has the versions as well) and compare to see if the version has changed.

We probably wouldn't want this to be a unit test, but rather a CLI command called in a CI workflow that runs only for our PRs. This way the PR CI check will fail and let the user know immediately that the rule needs to be minstacked.

Considered Alternatives

We can also exploring how to modify our internal logic to ignore when certain fields have changed so that we dont have to minstack our rules unecessarilily.

Additional Context

#4155

[shashank-elastic]

We already have a feature built in commad

❯ python -m detection_rules dev integrations show-latest-compatible --package okta --stack_version 8.13.0
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Compatible integration version=('3.2.0', ('There is a new integration okta version 3.2.0 available!', 'Update the rule min_stack version from 8.13.0 to 8.15.0 if using new features in this latest version.'))
(.venv) 
detection-rules on  issue-3195 [$!?] is 📦 v0.3.6 via 🐍 v3.12.5 (.venv) on ☁️  [email protected] 
❯ 

We could use this on minimum version supported! so , we will know whats the next applicable version, may be a checklist after we update the integration schema monthly! :)

[shashank-elastic]

This was reported to have the latest changes in integrations- https://github.com/elastic/integrations/pull/12154/files#diff-6fb787225d4f445c0e7645f4fcb538300d2d68724f30c9b17a77911ebfb06bcb.

When checked with the command

❯ python -m detection_rules dev integrations show-latest-compatible --package sentinel_one_cloud_funnel --stack_version 8.16.0
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Compatible integration version=('1.8.0', ('There is a new integration sentinel_one_cloud_funnel version 1.8.0 available!', 'Update the rule min_stack version from 8.16.0 to 8.16.2 if using new features in this latest version.'))

The helper command clearly suggests that we will min_stack only if new features are being used. So the changes for S1 packages are not filed related when checked.

We also ran a test lock versions command from todays latest main.

    - min_stack_version added: 8.13.0
  A: 0294f105-d7af-4a02-ae90-35f56763ffa2, new version: 204
    - min_stack_version added: 8.13.0
  A: 1e6363a6-3af5-41d4-b7ea-d475389c0ceb, new version: 106
    - min_stack_version added: 8.14.0
  A: d3551433-782f-4e22-bbea-c816af2d41c6, new version: 104
    - min_stack_version added: 8.14.0
  A: 095b6a58-8f88-4b59-827c-ab584ad4e759, new version: 204
    - min_stack_version added: 8.13.0
  A: 8a0fd93a-7df8-410d-8808-4cc5e340f2b9, new version: 204
    - min_stack_version added: 8.13.0
  A: 4030c951-448a-4017-a2da-ed60f6d14f4f, new version: 204
    - min_stack_version added: 8.13.0
  A: fc909baa-fb34-4c46-9691-be276ef4234c, new version: 204
    - min_stack_version added: 8.13.0
  A: 3af4cb9b-973f-4c54-be2b-7623c0e21b2b, new version: 204
    - min_stack_version added: 8.13.0
  A: 0e4367a0-a483-439d-ad2e-d90500b925fd, new version: 204
    - min_stack_version added: 8.13.0
  A: 41761cd3-380f-4d4d-89f3-46d6853ee35d, new version: 204
    - min_stack_version added: 8.13.0
  A: b483365c-98a8-40c0-92d8-0458ca25058a, new version: 105
    - min_stack_version added: 8.14.0
  A: 0abf0c5b-62dd-48d2-ac4e-6b43fe3a6e83, new version: 208
    - min_stack_version added: 8.14.0
  A: f59668de-caa0-4b84-94c1-3a1549e1e798, new version: 107
    - min_stack_version added: 8.14.0
  A: f94e898e-94f1-4545-8923-03e4b2866211, new version: 204
    - min_stack_version added: 8.13.0
  A: 61336fe6-c043-4743-ab6e-41292f439603, new version: 204
    - min_stack_version added: 8.13.0
  A: 846fe13f-6772-4c83-bd39-9d16d4ad1a81, new version: 107
    - min_stack_version added: 8.14.0
Updated /Users/shashankks/elastic_workspace/detection-rules/detection_rules/etc/version.lock.json file
Package saved to: /Users/shashankks/elastic_workspace/detection-rules/releases/8.18
loaded security_detection_engine manifests from the following package versions: ['8.17.1', '8.16.3', '8.16.2', '8.16.1', '8.15.12', '8.15.11', '8.15.10', '8.15.9', '8.15.8', '8.15.7', '8.15.6', '8.15.5', '8.15.4', '8.15.3', '8.15.2', '8.15.1', '8.14.18', '8.14.17', '8.14.16', '8.14.15', '8.14.14', '8.14.13', '8.14.12', '8.14.11', '8.14.10', '8.14.9', '8.14.8', '8.14.7', '8.14.6', '8.14.5', '8.14.4', '8.14.3', '8.14.2', '8.14.1', '8.13.23', '8.13.22', '8.13.21', '8.13.20', '8.13.19', '8.13.18', '8.13.17', '8.13.16', '8.13.15', '8.13.14', '8.13.13', '8.13.12', '8.13.11', '8.13.10', '8.13.9', '8.13.8', '8.13.7', '8.13.6', '8.13.5', '8.13.4', '8.13.3', '8.13.2', '8.13.1', '8.12.26', '8.12.25', '8.12.24', '8.12.23', '8.12.22', '8.12.21', '8.12.20', '8.12.19', '8.12.18', '8.12.17', '8.12.16', '8.12.15', '8.12.14', '8.12.13', '8.12.12', '8.12.11', '8.12.10', '8.12.9', '8.12.8', '8.12.7', '8.12.6', '8.12.5', '8.12.4', '8.12.3', '8.12.2', '8.12.1', '8.11.21', '8.11.20', '8.11.19', '8.11.18', '8.11.17', '8.11.16', '8.11.15', '8.11.14', '8.11.13', '8.11.12', '8.11.11', '8.11.10', '8.11.9', '8.11.8', '8.11.7', '8.11.6', '8.11.5', '8.11.4', '8.11.3', '8.11.2', '8.11.1', '8.10.18', '8.10.17', '8.10.16', '8.10.15', '8.10.14', '8.10.13', '8.10.12', '8.10.11', '8.10.10', '8.10.9', '8.10.8', '8.10.7', '8.10.6', '8.10.5', '8.10.4', '8.10.3', '8.10.2', '8.10.1', '8.9.15', '8.9.14', '8.9.13', '8.9.12', '8.9.11', '8.9.10', '8.9.9', '8.9.8', '8.9.7', '8.9.6', '8.9.5', '8.9.4', '8.9.3', '8.9.2', '8.9.1', '8.8.15', '8.8.14', '8.8.13', '8.8.12', '8.8.11', '8.8.10', '8.8.9', '8.8.8', '8.8.7', '8.8.6', '8.8.5', '8.8.4', '8.8.3', '8.8.2', '8.8.1', '8.7.13', '8.7.12', '8.7.11', '8.7.10', '8.7.9', '8.7.8', '8.7.7', '8.7.6', '8.7.5', '8.7.4', '8.7.3', '8.7.2', '8.7.1', '8.6.10', '8.6.9', '8.6.8', '8.6.7', '8.6.6', '8.6.5', '8.6.4', '8.6.3', '8.6.2', '8.6.1', '8.5.8', '8.5.7', '8.5.6', '8.5.5', '8.5.4', '8.5.3', '8.5.2', '8.5.1', '8.4.5', '8.4.4', '8.4.3', '8.4.2', '8.4.1', '8.3.4', '8.3.3', '8.3.2', '8.3.1', '8.2.1', '8.1.1', '1.0.2', '1.0.1']
[+] Adding all historical rule versions in our release package for version             8.18.0
[+] Adding historical rules from 8.17.1 package
- sha256: d630998bb58e88a8e7c598b03a6f4a92e9b3bc3b6f2a676f27d78eacc9ca0077
- 1279 rules included
diff saved to /Users/shashankks/elastic_workspace/detection-rules/lock-diff.txt
reverting changes in version.lock
Previous HEAD position was 419e5c1ad [Tuning] Suspicious WMI Event Subscription Created (#4327)
Switched to branch 'jan_schema_refresh'

The diff file is attached lock-diff.txt

In about 11 version modifications we never saw any double bumps. This should give us enough confidence that this may not result in double version bump tomo @w0rk3r, else we can proactively attend to the changes needed.

[shashank-elastic]

Look At the steps involved in analysis of minstack refer thread - https://elastic.slack.com/archives/C02USDK55AQ/p1736181401544619?thread_ts=1735946132.885459&cid=C02USDK55AQ

A straight forward CI can prove difficult in this case

[shashank-elastic]

16 Jan Update

I tried remodifying the code to fetch the schema of the current version and the previous version.

                                   rule_stack_version: Version,
                                   packages_manifest: dict) -> Union[None, Tuple[str, str]]:
    """Finds least compatible version for specified integration based on stack version supplied."""

    if not package:
        raise ValueError("Package must be specified")

    package_manifest = packages_manifest.get(package)
    schemas = load_integrations_schemas()
    if package_manifest is None:
        raise ValueError(f"Package {package} not found in manifest.")

    # Converts the dict keys (version numbers) to Version objects for proper sorting (descending)
    integration_manifests = sorted(package_manifest.items(), key=lambda x: Version.parse(x[0]), reverse=True)
    notice = ""

    for i, (version, manifest) in enumerate(integration_manifests):
        kibana_conditions = manifest.get("conditions", {}).get("kibana", {})
        version_requirement = kibana_conditions.get("version")
        if not version_requirement:
            raise ValueError(f"Manifest for {package}:{integration} version {version} is missing conditions.")

        compatible_versions = re.sub(r"\>|\<|\=|\^", "", version_requirement).split(" || ")

        if not compatible_versions:
            raise ValueError(f"Manifest for {package}:{integration} version {version} is missing compatible versions")

        highest_compatible_version = Version.parse(max(compatible_versions,
                                                       key=lambda x: Version.parse(x)))

        if highest_compatible_version > rule_stack_version:
            # generate notice message that a later integration version is available
            integration = f" {integration.strip()}" if integration else ""


            # Access the next version object
            next_version = integration_manifests[i + 1] if i + 1 < len(integration_manifests) else None
            current_schema = schemas[package][version]
            next_version_schema = schemas[package][next_version[0]]
            next_version_changes = compare_schemas(current_schema, next_version_schema)
            
            notice = (f"There is a new integration {package}{integration} version {version} available!",
                      f"Update the rule min_stack version from {rule_stack_version} to "
                      f"{highest_compatible_version} if using new features in this latest version."
                      f"Field changes when package version {version} and {next_version[0]} is compared: {next_version_changes}")
        if highest_compatible_version.major == rule_stack_version.major:
            return version, notice

        else:
            # Check for rules that cross majors
            for compatible_version in compatible_versions:
                if Version.parse(compatible_version) <= rule_stack_version:
                    return version, notice

    raise ValueError(f"no compatible version for integration {package}:{integration}")

def compare_schemas(previous_schema, current_schema):
    changes = []
    previous_fields = set(previous_schema.keys())
    current_fields = set(current_schema.keys())

    added_fields = current_fields - previous_fields
    removed_fields = previous_fields - current_fields
    common_fields = previous_fields & current_fields

    for field in added_fields:
        changes.append(f"Added field: {field}")

    for field in removed_fields:
        changes.append(f"Removed field: {field}")

    for field in common_fields:
        if previous_schema[field] != current_schema[field]:
            changes.append(f"Changed field: {field}")
    if changes:
        return "\n".join(changes)
    return "No Filed changes identified"

With the sample PR - https://github.com/elastic/integrations/pull/12285/files#diff-894c31b0555893a4caddf23f700a74a88de3c2663ff726d63f75e66b2ef403d5 where S2 has a new version say 1.8.1 i pulled in the latest schema and tried checking with this I was not able to pin point the latest field changes

Compatible integration version=('1.8.1', ('There is a new integration sentinel_one_cloud_funnel version 1.8.1 available!', 'Update the rule min_stack version from 8.13.0 to 8.16.2 if using new features in this latest version.Field changes when package version 1.8.1 and 1.8.0 is compared: No Filed changes identified'))

and also for some previous examples where we have min stacked

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Compatible integration version=('2.3.3', ('There is a new integration windows version 2.3.3 available!', 'Update the rule min_stack version from 8.13.0 to 8.14.0 if using new features in this latest version.Field changes when package version 2.3.3 and 2.3.2 is compared: No Filed changes identified'))
(.venv) 

Now when i think back after todays integration.

1 - Irrespective of the filed changes if the related integrations version requirement changes like below, we will ensure double bumps.

2. This is already being achieved by the current show-latest-compatible command in the package.
3. Do we need to drill down the fields, I don't think that would require.
4. We should probably refresh schema, run compatible versions for supported versions and analyse it for all packages and see the results.
5. When a minstack change is warranted we could see a CI usecase. Potentially this will be a slower CI model with having to load package manifests

[shashank-elastic]

16 Jan Update

So If you look at the journey of Okta Min-stack for currently supported and released versions.

We min stacked at 8.15.0 when schema was refreshed in December - #4290

 python -m detection_rules dev integrations show-latest-compatible  --package okta --stack_version 8.12.0
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Compatible integration version=('3.4.0', ('There is a new integration okta version 3.4.0 available!', 'Update the rule min_stack version from 8.12.0 to 8.15.0 if using new features in this latest version.'))
(.venv) 
detection-rules on  main [$?] is 📦 v0.3.17 via 🐍 v3.12.5 (.venv) on ☁️  [email protected] 
❯ python -m detection_rules dev integrations show-latest-compatible  --package okta --stack_version 8.13.0
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Compatible integration version=('3.4.0', ('There is a new integration okta version 3.4.0 available!', 'Update the rule min_stack version from 8.13.0 to 8.15.0 if using new features in this latest version.'))
(.venv) 
detection-rules on  main [$?] is 📦 v0.3.17 via 🐍 v3.12.5 (.venv) on ☁️  [email protected] 
❯ python -m detection_rules dev integrations show-latest-compatible  --package okta --stack_version 8.14.0
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Compatible integration version=('3.4.0', ('There is a new integration okta version 3.4.0 available!', 'Update the rule min_stack version from 8.14.0 to 8.15.0 if using new features in this latest version.'))
(.venv) 
detection-rules on  main [$?] is 📦 v0.3.17 via 🐍 v3.12.5 (.venv) on ☁️  [email protected] 
❯ python -m detection_rules dev integrations show-latest-compatible  --package okta --stack_version 8.15.0
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Compatible integration version=('3.4.0', '')
(.venv) 
detection-rules on  main [$?] is 📦 v0.3.17 via 🐍 v3.12.5 (.venv) on ☁️  [email protected] 
❯ python -m detection_rules dev integrations show-latest-compatible  --package okta --stack_version 8.16.0
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Compatible integration version=('3.4.0', '')
(.venv) 
detection-rules on  main [$?] is 📦 v0.3.17 via 🐍 v3.12.5 (.venv) on ☁️  [email protected] 
❯ python -m detection_rules dev integrations show-latest-compatible  --package okta --stack_version 8.17.0
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Compatible integration version=('3.4.0', '')
(.venv) 
detection-rules on  main [$?] is 📦 v0.3.17 via 🐍 v3.12.5 (.venv) on ☁️  [email protected] 
❯ 

The current refresh in January we have not seen any new versions making changes to stack version compatibility