[Rule Tuning] Add KEEP Command to all ES|QL Rules (#4146)

* updating ES|QL rules to include KEEP command

* fixed some ES|QL rules with typos; added validation for KEEP command

* fixed ES|QL errors from missing fields

* fixed flake errors

* updated date

* added best practices to hunt docs

Removed changes from:
- rules/cross-platform/execution_potential_widespread_malware_infection.toml
- rules/integrations/aws/impact_aws_s3_bucket_enumeration_or_brute_force.toml
- rules/integrations/aws/impact_s3_bucket_object_uploaded_with_ransom_extension.toml
- rules/integrations/aws/impact_s3_object_encryption_with_external_key.toml
- rules/integrations/aws/initial_access_signin_console_login_no_mfa.toml
- rules/integrations/aws/persistence_iam_user_created_access_keys_for_another_user.toml
- rules/integrations/aws/privilege_escalation_iam_administratoraccess_policy_attached_to_group.toml
- rules/integrations/aws/privilege_escalation_iam_administratoraccess_policy_attached_to_role.toml
- rules/integrations/aws/privilege_escalation_iam_administratoraccess_policy_attached_to_user.toml
- rules/integrations/aws_bedrock/aws_bedrock_guardrails_multiple_violations_by_single_user.toml
- rules/integrations/aws_bedrock/aws_bedrock_guardrails_multiple_violations_in_single_request.toml
- rules/integrations/aws_bedrock/aws_bedrock_high_confidence_misconduct_blocks_detected.toml
- rules/integrations/aws_bedrock/aws_bedrock_high_resource_consumption_detection.toml
- rules/integrations/aws_bedrock/aws_bedrock_multiple_attempts_to_use_denied_models_by_user.toml
- rules/integrations/aws_bedrock/aws_bedrock_multiple_validation_exception_errors_by_single_user.toml
- rules/integrations/azure/credential_access_entra_signin_brute_force_microsoft_365.toml
- rules/integrations/azure/credential_access_entra_signin_brute_force_microsoft_365_repeat_source.toml
- rules/integrations/o365/credential_access_microsoft_365_brute_force_user_account_attempt.toml
- rules/integrations/okta/credential_access_multiple_device_token_hashes_for_single_okta_session.toml
- rules/integrations/okta/credential_access_okta_authentication_for_multiple_users_from_single_source.toml
- rules/integrations/okta/credential_access_okta_authentication_for_multiple_users_with_the_same_device_token_hash.toml
- rules/integrations/okta/credential_access_okta_multiple_device_token_hashes_for_single_user.toml
- rules/integrations/okta/initial_access_okta_user_sessions_started_from_different_geolocations.toml

(selectively cherry picked from commit 06319b7)

detection_rules/rule.py

@@ -926,6 +926,13 @@ def validates_esql_data(self, data, **kwargs):
                 f" Add 'metadata _id, _version, _index' to the from command or add an aggregate function."
             )
 
+        # Enforce KEEP command for ESQL rules
+        if '| keep' not in query_lower:
+            raise ValidationError(
+                f"Rule: {data['name']} does not contain a 'keep' command ->"
+                f" Add a 'keep' command to the query."
+            )
+
 
 @dataclass(frozen=True)
 class ThreatMatchRuleData(QueryRuleData):

hunting/README.md

@@ -44,6 +44,12 @@ Otherwise, the names do not require the integration, since it is already annotat
   - **mitre**: Reference to applicable MITRE ATT&CK tactics or techniques that the rule addresses, enhancing the contextual understanding of its security implications.
   - **references**: Links to external documents, research papers, or websites that provide additional information or validation for the detection logic.
 
+#### Query Best Practices
+* Use `KEEP` command to select specific fields that are relevant or necessary for `STATS` command
+* Use `LIMIT` command to limit the number of results, depending on expected result volume
+* Filter as much as possible in `WHERE` command to reduce events needed to be processed
+* For `FROM` command for index patterns, be as specific as possible to reduce potential event matches that are irrelevant
+
 ### Field Usage
 Use standardized fields where possible to ensure that queries are compatible across different data environments and sources.
 

rules/integrations/aws/discovery_ec2_multi_region_describe_instances.toml

@@ -1,7 +1,7 @@
 [metadata]
 creation_date = "2024/08/26"
 maturity = "production"
-updated_date = "2024/08/26"
+updated_date = "2024/10/09"
 
 [rule]
 author = ["Elastic"]
@@ -41,6 +41,9 @@ from logs-aws.cloudtrail-*
 // truncate the timestamp to a 30-second window
 | eval target_time_window = DATE_TRUNC(30 seconds, @timestamp)
 
+// keep only the relevant fields
+| keep target_time_window, aws.cloudtrail.user_identity.arn, cloud.region
+
 // count the number of unique regions and total API calls within the 30-second window
 | stats region_count = count_distinct(cloud.region), window_count = count(*) by target_time_window, aws.cloudtrail.user_identity.arn
 

rules/integrations/aws/discovery_servicequotas_multi_region_service_quota_requests.toml

@@ -1,7 +1,7 @@
 [metadata]
 creation_date = "2024/08/26"
 maturity = "production"
-updated_date = "2024/10/02"
+updated_date = "2024/10/09"
 
 [rule]
 author = ["Elastic"]
@@ -48,6 +48,9 @@ from logs-aws.cloudtrail-*
 // filter for EC2 service quota L-1216C47A (vCPU on-demand instances)
 | where service_code == "ec2" and quota_code == "L-1216C47A"
 
+// keep only the relevant fields
+| keep target_time_window, aws.cloudtrail.user_identity.arn, cloud.region, service_code, quota_code
+
 // count the number of unique regions and total API calls within the 30-second window
 | stats region_count = count_distinct(cloud.region), window_count = count(*) by target_time_window, aws.cloudtrail.user_identity.arn