[FR] Support new_terms schema import/export w/custom format

[Mikaayenson]

Summary

Related to #3542

This PR supports import/export of the new_terms rule type w/custom toml formatting.

In #2360 We added the inital support to new_terms schemas, however this introduced something new to our ruleset where we added custom formating in our toml files that does not 1-1 match the schema exported from Kibana. The custom formatting within the toml rule IMO is more organized in this way. The original intent also was to manually develop rules for prebuilt content and push externally (one-way support).

The problem is that it introduced an issue where it prevented us from importing the rules from ndjson since the schemas do not much (two-way support of import/export).

In #3569 we introduced a stop-gap to the cli_utils that allowed us to also import new_term rules in our schema using the rule prompt. The intent was to wait until we could refactor the new_terms rules and update the schemas to match 1-1 with kibana exports.

This PR on the other hand maintains the current format and instead adds an pre_load to parse the ndjson and format into toml rules.

The benefit is that we should be able to keep the same toml format, which means we also shouldnt have to update rules at all.

Note: If after thorough testing, this still doesn't meet our needs, we can still update the schemas so that the perfectly match 1-1 at a later time & close (not merge).

Testing

• Try to import rules from kibana (on main using the kibana).
• I tested this on the DAC-feature with custom rules.
• Import / export should work on the core detection-rules and kibana cli commands.

Additional Info

Cleaning up a couple bugs while testing different features with import/export rules:

• 2ef6fe8
• b5e1630

[eric-forte-elastic]

---

Update: Issue below is unrelated to this PR, no reason to block

🟢 Testing looks good 👍 For import and export (still need to test to/from repo commands)

Issue is with the call to toml_write on line 1355 of rule.py, it adds a newline where it should not. If one replaces it with the following, it does not make the same mistake.

with self.path.open("w") as f:
    pytoml.dump(converted, f)

Output with Fixed Formatting

detection-rules on  new_terms_schema_preload [?] is  v0.1.0 via  v3.12.4 (detection-rules-build) on  eric.forte took 2s 
❯ python -m detection_rules kibana import-rules -f test_dir/discovery_enumeration_of_privileged_local_groups_membership_duplicate.toml -o
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

1 rule(s) successfully imported
 - 92bf24a6-3aad-49ce-8bce-cd64391afe4a

---

[OLD] 🟡 Testing [OLD]

• Testing from Kibana exporting to TOML new terms works correctly, but importing back to Kibana currently throws an Toml parsing error for me.

Looking into it now, but for some reason when the TOML is written the rule.filters.meta is written as:

value = '{"wildcard":{"winlog.event_data.CallerProcessName":{"case_insensitive":true,"value":"C:\\\\Program Files\\\\*.exe"}}}
'

Instead the following which does parse:

value = '{"wildcard":{"winlog.event_data.CallerProcessName":{"case_insensitive":true,"value":"C:\\\\Program Files\\\\*.exe"}}}'

May be unrelated to the additions in this PR, as fixing this formatting issue does allow the rule to successfully import.

Test Rule Example

[metadata]
creation_date = "2024/07/12"
maturity = "production"
updated_date = "2024/07/12"

[rule]
actions = []
author = ["Elastic"]
description = """
Identifies instances of an unusual process enumerating built-in Windows privileged local groups membership like
Administrators or Remote Desktop users.
"""
enabled = false
exceptions_list = []
false_positives = []
from = "now-540s"
index = ["winlogbeat-*", "logs-system.*", "logs-windows.*"]
interval = "5m"
language = "kuery"
license = "Elastic License v2"
max_signals = 100
name = "Enumeration of Privileged Local Groups Membership [Duplicate]"
note = """## Triage and analysis

### Investigating Enumeration of Privileged Local Groups Membership

After successfully compromising an environment, attackers may try to gain situational awareness to plan their next steps. This can happen by running commands to enumerate network resources, users, connections, files, and installed security software.

This rule looks for the enumeration of privileged local groups' membership by suspicious processes, and excludes known legitimate utilities and programs installed. Attackers can use this information to decide the next steps of the attack, such as mapping targets for credential compromise and other post-exploitation activities.

> **Note**:
> This investigation guide uses the [Osquery Markdown Plugin](https://www.elastic.co/guide/en/security/master/invest-guide-run-osquery.html) introduced in Elastic Stack version 8.5.0. Older Elastic Stack versions will display unrendered Markdown in this guide.

#### Possible investigation steps

- Identify the process, host and user involved on the event.
- Investigate the process execution chain (parent process tree) for unknown processes. Examine their executable files for prevalence, whether they are located in expected locations, and if they are signed with valid digital signatures.
- Investigate other alerts associated with the user/host during the past 48 hours.
- Investigate any abnormal account behavior, such as command executions, file creations or modifications, and network connections.
- Examine the host for derived artifacts that indicate suspicious activities:
  - Analyze the process executable using a private sandboxed analysis system.
  - Observe and collect information about the following activities in both the sandbox and the alert subject host:
    - Attempts to contact external domains and addresses.
      - Use the Elastic Defend network events to determine domains and addresses contacted by the subject process by filtering by the process' `process.entity_id`.
      - Examine the DNS cache for suspicious or anomalous entries.
        - !{osquery{"label":"Osquery - Retrieve DNS Cache","query":"SELECT * FROM dns_cache"}}
    - Use the Elastic Defend registry events to examine registry keys accessed, modified, or created by the related processes in the process tree.
    - Examine the host services for suspicious or anomalous entries.
      - !{osquery{"label":"Osquery - Retrieve All Services","query":"SELECT description, display_name, name, path, pid, service_type, start_type, status, user_account FROM services"}}
      - !{osquery{"label":"Osquery - Retrieve Services Running on User Accounts","query":"SELECT description, display_name, name, path, pid, service_type, start_type, status, user_account FROM services WHERE\\nNOT (user_account LIKE '%LocalSystem' OR user_account LIKE '%LocalService' OR user_account LIKE '%NetworkService' OR\\nuser_account == null)\\n"}}
      - !{osquery{"label":"Osquery - Retrieve Service Unsigned Executables with Virustotal Link","query":"SELECT concat('https://www.virustotal.com/gui/file/', sha1) AS VtLink, name, description, start_type, status, pid,\\nservices.path FROM services JOIN authenticode ON services.path = authenticode.path OR services.module_path =\\nauthenticode.path JOIN hash ON services.path = hash.path WHERE authenticode.result != 'trusted'\\n"}}
  - Retrieve the files' SHA-256 hash values using the PowerShell `Get-FileHash` cmdlet and search for the existence and reputation of the hashes in resources like VirusTotal, Hybrid-Analysis, CISCO Talos, Any.run, etc.
- Investigate potentially compromised accounts. Analysts can do this by searching for login events (for example, 4624) to the target host after the registry modification.

### False positive analysis

- Discovery activities are not inherently malicious if they occur in isolation. As long as the analyst did not identify suspicious activity related to the user or host, such alerts can be dismissed.
- If this rule is noisy in your environment due to expected activity, consider adding exceptions — preferably with a combination of user and command line conditions.

### Response and remediation

- Initiate the incident response process based on the outcome of the triage.
- Isolate the involved hosts to prevent further post-compromise behavior.
- Investigate credential exposure on systems compromised or used by the attacker to ensure all compromised accounts are identified. Reset passwords for these accounts and other potentially compromised credentials, such as email, business systems, and web services.
- Run a full antimalware scan. This may reveal additional artifacts left in the system, persistence mechanisms, and malware components.
- Determine the initial vector abused by the attacker and take action to prevent reinfection through the same vector.
- Using the incident response data, update logging and audit policies to improve the mean time to detect (MTTD) and the mean time to respond (MTTR).
"""
references = []
related_integrations = []
required_fields = []
risk_score = 47
risk_score_mapping = []
rule_id = "92bf24a6-3aad-49ce-8bce-cd64391afe4a"
setup = ""
severity = "medium"
severity_mapping = []
tags = [
    "Domain: Endpoint",
    "OS: Windows",
    "Use Case: Threat Detection",
    "Tactic: Discovery",
    "Resources: Investigation Guide",
]
timestamp_override = "event.ingested"
to = "now"
type = "new_terms"

query = '''
host.os.type:windows and event.category:iam and event.action:user-member-enumerated and 
  (
    group.name:(*Admin* or "RemoteDesktopUsers") or
    winlog.event_data.TargetSid:("S-1-5-32-544" or "S-1-5-32-555")
  ) and 
  not (
    winlog.event_data.SubjectUserName: *$ or
    winlog.event_data.SubjectUserSid: ("S-1-5-19" or "S-1-5-20") or 
    winlog.event_data.CallerProcessName:("-" or 
                                       C\:\\Windows\\System32\\VSSVC.exe or 
                                       C\:\\Windows\\System32\\SearchIndexer.exe or 
                                       C\:\\Windows\\System32\\CompatTelRunner.exe or 
                                       C\:\\Windows\\System32\\oobe\\msoobe.exe or
                                       C\:\\Windows\\System32\\net1.exe or 
                                       C\:\\Windows\\System32\\svchost.exe or 
                                       C\:\\Windows\\System32\\Netplwiz.exe or 
                                       C\:\\Windows\\System32\\msiexec.exe or
                                       C\:\\Windows\\System32\\CloudExperienceHostBroker.exe or
                                       C\:\\Windows\\System32\\RuntimeBroker.exe or
                                       C\:\\Windows\\System32\\wbem\\WmiPrvSE.exe or
                                       C\:\\Windows\\System32\\SrTasks.exe or
                                       C\:\\Windows\\System32\\diskshadow.exe or
                                       C\:\\Windows\\System32\\dfsrs.exe or
                                       C\:\\Windows\\System32\\vssadmin.exe or
                                       C\:\\Windows\\System32\\dllhost.exe or
                                       C\:\\Windows\\System32\\mmc.exe or
                                       C\:\\Windows\\System32\\SettingSyncHost.exe or
                                       C\:\\Windows\\System32\\inetsrv\\w3wp.exe or
                                       C\:\\Windows\\System32\\wsmprovhost.exe or
                                       C\:\\Windows\\System32\\mstsc.exe or
                                       C\:\\Windows\\System32\\esentutl.exe or
                                       C\:\\Windows\\System32\\RecoveryDrive.exe or
                                       C\:\\Windows\\System32\\SystemPropertiesComputerName.exe or
                                       C\:\\Windows\\SysWOW64\\msiexec.exe or
                                       C\:\\Windows\\ImmersiveControlPanel\\SystemSettings.exe or
                                       C\:\\Windows\\Temp\\rubrik_vmware*\\snaptool.exe or
                                       C\:\\Windows\\VeeamVssSupport\\VeeamGuestHelper.exe or
                                       C\:\\WindowsAzure\\*WaAppAgent.exe or
                                       C\:\\$WINDOWS.~BT\\Sources\\*.exe
                                      )
  )
'''


[[rule.filters]]

[rule.filters."$state"]
store = "appState"
[rule.filters.meta]
disabled = false
key = "query"
negate = true
type = "custom"
value = """
{"wildcard":{"winlog.event_data.CallerProcessName":{"case_insensitive":true,"value":"C:\\\\Program Files
(x86)\\\\*.exe"}}}
"""
[rule.filters.query.wildcard."winlog.event_data.CallerProcessName"]
case_insensitive = true
value = "C:\\\\Program Files (x86)\\\\*.exe"
[[rule.filters]]

[rule.filters."$state"]
store = "appState"
[rule.filters.meta]
disabled = false
key = "query"
negate = true
type = "custom"
value = '{"wildcard":{"winlog.event_data.CallerProcessName":{"case_insensitive":true,"value":"C:\\\\Program Files\\\\*.exe"}}}
'
[rule.filters.query.wildcard."winlog.event_data.CallerProcessName"]
case_insensitive = true
value = "C:\\\\Program Files\\\\*.exe"
[[rule.threat]]
framework = "MITRE ATT&CK"
[[rule.threat.technique]]
id = "T1069"
name = "Permission Groups Discovery"
reference = "https://attack.mitre.org/techniques/T1069/"
[[rule.threat.technique.subtechnique]]
id = "T1069.001"
name = "Local Groups"
reference = "https://attack.mitre.org/techniques/T1069/001/"



[rule.threat.tactic]
id = "TA0007"
name = "Discovery"
reference = "https://attack.mitre.org/tactics/TA0007/"

[rule.meta]
from = "4m"
kibana_siem_app_url = ""

[rule.new_terms]
field = "new_terms_fields"
value = ["host.id", "winlog.event_data.SubjectUserName", "winlog.event_data.CallerProcessName"]
[[rule.new_terms.history_window_start]]
field = "history_window_start"
value = "now-14d"

Details

detection-rules on  new_terms_schema_preload [?] is  v0.1.0 via  v3.12.4 (detection-rules-build) on  eric.forte 
❯ python -m detection_rules kibana export-rules -d test_dir
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

18 rules exported
18 rules converted
18 saved to test_dir

detection-rules on  new_terms_schema_preload [?] is  v0.1.0 via  v3.12.4 (detection-rules-build) on  eric.forte took 3s 
❯ python -m detection_rules kibana import-rules -f test_dir/discovery_enumeration_of_privileged_local_groups_membership_duplicate.toml -o
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Error loading rule in /home/forteea1/Code/clean_mains/detection-rules/test_dir/discovery_enumeration_of_privileged_local_groups_membership_duplicate.toml
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/__main__.py", line 35, in <module>
    main()
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/__main__.py", line 32, in main
    root(prog_name="detection_rules")
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/cli_utils.py", line 82, in get_collection
    rules.load_files(Path(p) for p in rule_files)
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/rule_loader.py", line 487, in load_files
    self.load_file(path)
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/rule_loader.py", line 434, in load_file
    obj = self._load_toml_file(path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/rule_loader.py", line 363, in _load_toml_file
    toml_dict = self.deserialize_toml_string(f.read())
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/rule_loader.py", line 353, in deserialize_toml_string
    return pytoml.loads(contents)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/pytoml/parser.py", line 24, in loads
    ast = _p_toml(src, object_pairs_hook=object_pairs_hook)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/pytoml/parser.py", line 341, in _p_toml
    s.expect_eof()
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/pytoml/parser.py", line 123, in expect_eof
    return self._expect(self.consume_eof())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/pytoml/parser.py", line 163, in _expect
    raise TomlError('msg', self._pos[0], self._pos[1], self._filename)
pytoml.core.TomlError: <string>(159, 1): msg

[eric-forte-elastic]

🟢 Testing looks good for export-rule-from-repo

Details

[Mikaayenson]

fix: default formatter inadvertently adds a newline

[eric-forte-elastic]

🟢 Great peer review, looks good to me! 👍

[Mikaayenson]

@eric-forte-elastic Thanks for the verbose testing / debug session 🥳

[eric-forte-elastic]

Found something (sorry trying to remove approval cleanly my bad)

[Mikaayenson]

Found something (sorry trying to remove approval cleanly my bad)

Sure thing. lets dive in

[eric-forte-elastic]

Peer review fixed issue 👍

[Mikaayenson]

Peer review fixed issue 👍

++ so it turns out there was indeed a bug (introduced in a prior PR).

[Mikaayenson]

fix: opaque issue where the contents is in a different format depending on how the rule is loaded (kibana or from toml)