As part of our ongoing commitment to customers, AWS is providing this security incident response playbook that describes the steps needed to detect and respond to compromised credentials within your AWS account(s). The purpose of this document is to provide prescriptive guidance on the actions to take once you suspect a security event has taken place.
Aspects of AWS incident response
In order to quickly and effectively execute incident response activities, it is crucial to prepare the people, processes, and technology within your organization. Review the AWS Security Incident Response Guide, and implement the steps required to help ensure preparedness of all three domains.
It is important that you notify AWS as soon as you suspect a compromised credential(s) within your AWS account or Organization. Here are the steps to engage AWS Support:
-
Login to your AWS account:
- This is the first AWS account that was impacted by the security event, to validate AWS account ownership.
- Note: If you are unable to access account, use this form to submit a support request.
-
Select “Services”, “Support Center”, “Create Case”.
-
Select issue type “Account and Billing”.
-
Select the impacted Service and Category:
- Example:
- Service: Account
- Category: Security
- Example:
-
Choose a Severity:
- Enterprise Support or On-Ramp Customers: Critical Business Risk Question.
- Business Support Customers: Urgent Business Risk Question.
-
Describe your question or issue:
- Provide a detailed description of the security issue you’re experiencing, the impacted resources, and the business impact.
- Time the security event was first recognized.
- Summary of the event timeline to this point.
- Artifacts you’ve collected (screenshots, log files).
- ARN of IAM Users or Roles that you suspect are compromised.
- Description of affected resources, and the business impact.
- Level of engagement in your organization (ex: “This security event has the visibility of the CEO and Board of Directors”).
- Optional: Add additional case contacts.
- Provide a detailed description of the security issue you’re experiencing, the impacted resources, and the business impact.
-
Select “Contact Us”.
- Preferred contact language (may be subject to availability)
- Preferred contact method: Web, Phone, or Chat (recommended)
- Optional: Additional case contacts
- Click “Submit”
-
Escalations: Please notify your AWS account team as soon as possible, so they may engage the necessary resources and escalate as needed.
Note: It is very important to verify your 'Alternate Security Contact' is defined for each AWS account. For more details, please refer to this article.
There are multiple ways to detect compromised credentials within your AWS environment. Listed are some ways to detect potential Indicators of Compromise (IoC). For a detailed list of IoC’s please refer to appendix A. Note: Document the details of each suspected IoC for further analysis.
-
Review Amazon GuardDuty findings. The findings below are related to suspicious or anomalous activity by IAM entities. Review the finding details, and if the activity is unexpected, this could indicate a compromised credential.
- CredentialAccess:IAMUser/AnomalousBehavior
- DefenseEvasion:IAMUser/AnomalousBehavior
- Discovery:IAMUser/AnomalousBehavior
- Exfiltration:IAMUser/AnomalousBehavior
- Impact:IAMUser/AnomalousBehavior
- InitialAccess:IAMUser/AnomalousBehavior
- PenTest:IAMUser/KaliLinux
- PenTest:IAMUser/ParrotLinux
- PenTest:IAMUser/PentooLinux
- Persistence:IAMUser/AnomalousBehavior
- Policy:IAMUser/RootCredentialUsage
- PrivilegeEscalation:IAMUser/AnomalousBehavior
- Recon:IAMUser/MaliciousIPCaller
- Recon:IAMUser/MaliciousIPCaller.Custom
- Recon:IAMUser/TorIPCaller
- Stealth:IAMUser/CloudTrailLoggingDisabled
- Stealth:IAMUser/PasswordPolicyChange
- UnauthorizedAccess:IAMUser/ConsoleLoginSuccess.B
- UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration
- UnauthorizedAccess:IAMUser/MaliciousIPCaller
- UnauthorizedAccess:IAMUser/MaliciousIPCaller.Custom
- UnauthorizedAccess:IAMUser/TorIPCaller
-
Review AWS Billing for unusual spikes which can be a sign of account and credential(s) compromise by following these steps:
- Sign into the AWS Management Console and open the Billing console.
- Choose “Bills” to see details about your current charges.
- Choose “Payments” to see your historical payment transactions.
- Choose “AWS Cost and Usage Reports” to see reports that break down your costs.
-
Download the IAM credentials report by going to the IAM console and choose the “Credential report” on the left under “Access reports” then review the following:
- identify unusual IAM user creation by looking at creation date and the password last used/changed columns.
- Check if any IAM users have two or more access keys.
- Check if any IAM users have AWSCompromisedKeyQuarantineV2 attached. If so, rotate its access keys.
-
Review IAM Roles within the AWS account, to identify any unfamiliar roles that have been created or accessed
-
Within the AWS Console, select “Services”, “IAM”, and “Roles”
-
Review any unfamiliar “Role names”.
-
Click on the Role name, and review the listed details:
- Role Creation Date
- ARN
- Last Activity
-
Click “Permissions” to review IAM Policies that are attached to the role.
-
Click “Trust Relationships” to view entities that can assume the role.
-
Click “Access Advisor” to show which services have been accessed by the role, and the “Last Accessed Date”.
-
-
Detect any unrecognized or unauthorized resources within your AWS account such as:
- EC2 instances by running the following command in the AWS CLI terminal “aws ec2 describe-instances” and validate the results. Look for the creation time for each instance using the --query 'Reservations[].Instances[].{ip: PublicIpAddress, tm: LaunchTime}' --filters 'Name=tag:Name,Values= myInstanceName' | jq 'sort_by(.tm) | reverse | .[0]' filter.
- Lambda functions by running the following command in the AWS CLI terminal “aws lambda list-functions” and validate the last modified time by using the “| grep “LastModified”” filter.
-
Review any security notifications from AWS about your account by signing into AWS Support Center then reading and responding to the message(s).
-
Review findings by following these steps:
- Open the IAM console.
- Choose “Access analyzer” from the left column under Access reports.
- Under “Active findings” review findings to identify resources in your organization, such as S3 buckets, IAM roles, or Lambda functions that are shared with external identities.
Once you have identified any suspicious resources or activity that may indicate compromise, perform further analysis in your SIEM, or log analysis tools. AWS has various security services tools to aid in analyzing security events. Some of these tools include:
-
Amazon Guardduty - Amazon GuardDuty is a threat detection service that continuously monitors for malicious activity and unauthorized behavior to protect your AWS accounts, Amazon Elastic Compute Cloud (EC2) workloads, container applications, Amazon Aurora databases, and data stored in Amazon Simple Storage Service (S3).
-
AWS Security Hub - AWS Security Hub is a cloud security posture management (CSPM) service that performs automated, continuous security best practice checks against your AWS resources to help you identify misconfigurations, and aggregates your security alerts (i.e. findings) in a standardized format so that you can more easily enrich, investigate, and remediate them.
-
Amazon Detective - Amazon Detective makes it easier to analyze, investigate, and quickly identify the root cause of potential security issues or suspicious activities.
You can also search your AWS CloudTrail event history, by following these steps to analyze and gather evidence:
-
Analyze AWS CloudTrail logs for the following:
- Any unusual activities associated with logins:
- Go to the CloudTrail Dashboard.
- In the left-hand side, pick Event History.
- In the dropdown menu change Read-Only to Event Name.
- Review the available events for any suspicious activity by searching for the following terms via the search box: “ConsoleLogin”, “AssumeRole”, “GetFederationToken”, “GetCredentialReport”, “GenerateCredentialReport”. Also it is important to note that “userIdentity” should show up as "type": "Root" for the root user or "type": "IAMUser" for any local IAM users on the account.
- Locate any IAM access key ID and user name used to launch a suspicious Amazon EC2 instance:
- Open the AWS CloudTrail console and choose “Event history” from the navigation pane.
- Select the “Lookup attributes” dropdown menu, then choose “Resource name”.
- In the Enter resource name field, paste the EC2 instance ID, and then hit enter on your device.
- Expand the Event name for RunInstances.
- Copy the AWS access key, and note the User name.
- Review AWS CloudTrail event history for activity by the compromised access key:
- Open the CloudTrail console and choose “Event > history” from the navigation pane.
- Select the “Lookup attributes” dropdown menu, then > choose “AWS access key”.
- In the “Enter an AWS access key field”, enter the > compromised IAM access key ID.
- Expand the Event name for the API call RunInstances.
- If you are accessing AWS via 3rd Party Identity Provider, be sure to audit the access logs and follow that vendors guidance on responding to an event and securing your environment.
- Any unusual activities associated with logins:
-
Analyze IAM Access Advisor compromised credentials last accessed information to see what was the last service it access by following these steps:
- Go to the IAM console.
- Go to users or roles, click on the name of the compromised principal, choose the “access advisor” tab and look at which resources it last accessed.
After analyzing and gathering more information about the compromised credential(s) and any other affected resources, it’s time to control and suppress the security event.
-
Disable the IAM user(s), create a backup IAM access key, and then disable the compromised access key by following these steps:
- Open the IAM console and paste the IAM access key ID in the Search IAM bar.
- Choose the user name and then choose the “Security credentials” tab.
- In the Console password field, choose “Manage”.
- In Console access, choose “Disable”, and then select Apply.
- For the compromised IAM access key, choose “Make inactive”.
-
Rotate access keys by following these steps:
- First, create a second key by going to the IAM console.
- In the navigation pane, choose “Users”.
- Choose the name of the intended user, and then choose the “Security credentials” tab.
- In the “Access keys” section, choose “Create access key”. On the “Access key best practices & alternatives” page, choose “Other”, then choose “Next”.
- Then, modify your application to use the new key.
- Disable (but do not delete) the first key.
- If there are any problems with your application, reactivate the key temporarily. When your application is fully functional, and the first key is disabled, only then is it safe to delete the first key. Make sure you keep a record of all deleted access keys to continue searching for them in AWS CloudTrail logs.
-
Revoke the IAM role(s) active sessions by following these steps:
- Open the IAM console and go to role and click on the IAM role you want to revoke active sessions for.
- Click on the IAM role name and go to “revoke sessions” tab.
- Click on “revoke active sessions” button and confirm the step.
-
Isolate affected resources by following these steps:
- For Amazon EC2 instances, go to the EC2 instance console, check the box next to the EC2 instance you want to isolate, click on actions, click on security, then click on change security groups. Detach any current security groups and attach an isolated security group that blocks inbound and outbound communication to the EC2.
- For Amazon S3 buckets, use bucket policies to block any suspicious IP addresses from accessing the S3 buckets.
-
Revoke Identity Center sessions:
- With Identity Center, there are two sessions that to be concerned about which are the access portal session, and role/application sessions:
-
Access portal session:
- Disable the user in Identity Center:
- Navigate to the Identity Center console and select ‘Users’
- Choose the username of the user being disabled
- In the General information box for the user, click ‘Disable > user access’
- Revoke any active sessions:
- On the user’s page in Identity Center, select ‘Active > sessions’ tab
- Select any listed sessions and then click ‘Delete session’
- Disable the user in Identity Center:
-
Revoke role sessions:
- Use one of the two options proposed:
-
Identify the permission set(s) being used by the user.
-
From the Identity Center console, click on Permission Sets
-
Select the name of the permission set.
-
Scroll down to ‘Inline Policy’ and click on the Edit button
-
Add the following policy:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Deny", "Action": "*", "Resource": "*", "Condition": { "StringEquals": { "identitystore:userId": "example" }, "DateLessThan": { "aws:TokenIssueTime": "2023-09-26T15:00:00.000Z" } } } ] }
-
For this policy, update ‘example’ to the user’s Identity Center User ID. The User ID can be found in the ‘General information’ box of the user’s User page. The value for aws:TokenIssueTime should equal the time in which you apply this policy.
-
-
Alternatively, use a Service Control Policy to deny all actions for a specific user.
-
From the AWS management console, navigate to AWS Organizations.
-
Click on ‘Policies’ then select ‘Service Control Policies’.
-
Click ‘Create policy’ and add the following policy:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Deny", "Action": [ "*" ], "Resource": "*", "Condition": { "StringEquals": { "identitystore:userId": "Add user ID here" } } } ] }
-
Replace ‘Add user ID here’ with the user’s Identity Center User ID found in the ‘General information’ box of the user’s User page.
-
Attach this policy to the Root or a specific Organizational Unit where the user resides:
- Go back to the ‘Service Control Policies’ page.
- Select the newly created policy.
- Click on ‘Attach’.
- Choose the target Root or Organizational Unit to apply this policy.
-
-
Application sessions
Application sessions are created when a user accesses a third-party application or AWS service directly from the access portal.
To revoke application sessions, review the documentation of the application that was accessed.
- Compromised Identity Center
If your Identity provider is compromised, you should block access to Identity Center from a compromised identity provider (i.e. external SAML-based Identity Provider or Active Directory) by changing the identity source to ‘Identity Source directory’.
To change your identity source: 1. Navigate to the Identity Center console 2. Select ‘Settings’ 3. On the Settings page, click on the ‘Identity source’ tab 4. Click on the ‘Actions’ button and then select ‘Change identity source’ 5. Select ‘Identity Center directory’ on the ‘Choose identity source’ page 6. Click on the ‘Next’ button 7. Read the information contained in the ‘Review and confirm’ box, 8. Type ‘ACCEPT’ in the appropriate field, and click on the ‘Change identity source’ button
Please note, If you are accessing AWS via 3rd Party Identity Provider, be sure to audit the access logs and follow that vendors guidance on responding to an event and securing your environment.
Also, If you are changing from Active Directory to the Identity Center directory, all users and groups will be deleted. Permission sets will not be deleted, but permission set assignments will be deleted. If you are changing from an external SAML-based identity provider, all users and groups will remain populated in Identity Center as will the permission set assignments.
Once you are done with containing the security event, it’s time to work on removing and cleaning the cause of the security event.
-
Remove any resources created by the compromised key(s) which were detected in the “Analysis phase step 1”. Check all AWS regions, even regions where you do not launch AWS resources in. If you need to keep a resource for investigation, consider backing them up.
-
Check for and remove any unrecognized services that are running within your account. Pay special attention to the following resources:
- EC2 instances and AMIs, including instances in the stopped state.
- EBS volumes and snapshots.
- AWS Lambda functions and layers.
-
Remove unnecessary permissions from logging S3 buckets or log aggregation resources identified in “Detection phase step 6” that could be used to avoid detection. This lets you identify unintended access to your resources and data.
-
Remove any exposed data that is not needed for operations.
-
Perform security vulnerability scans on the publicly facing resources. You can use tools like Amazon Inspector to scan OS and 3rd party software application for vulnerabilities.
Once you are done with eradicating the security event causes. It’s time to restore the affected resources into a good-known state.
-
Restore necessary data from known clean backups that predate the event:
- Restoring from an Amazon EBS snapshot or an AMI.
- Restoring from a DB snapshot Amazon RDS.
- Restoring previous versions Amazon S3 object versions.
-
If necessary, rebuild systems from scratch, including redeploying from trusted source using automation, sometime in a new AWS account.
-
If access to Multi Factor Authentication “MFA” is lost and you don’t have another MFA device registered to you’re root account , please follow these steps:
- Sign in to the AWS Management Console as the account owner by choosing Root user and entering your AWS account email address. On the next page, enter your password.
- On the Additional verification required page, select an MFA method to authenticate with and choose Next.
- Depending on the type of MFA you are using, you should see a different page, but the “Troubleshoot MFA” option functions the same. On the “Additional verification required” page or “Multi-factor authentication” page, choose “Troubleshoot MFA”.
- If required, type your password again and choose Sign in.
- On the “Troubleshoot your authentication device” page, in the “Sign in using alternative factors of authentication” section, choose “Sign in using alternative factors”.
- On the “Sign in using alternative factor"s of authentication” page, authenticate your account by verifying the email address, choose “Send verification email”
- Check the email that is associated with your AWS account for a message from Amazon Web Services ([email protected]). Follow the directions in the email. If you don't see the email in your account, check your spam folder, or return to your browser and choose “Resend the email”.
-
After you verify your email address, you can continue authenticating your account. To verify your primary contact phone number, choose Call me now.
-
Answer the call from AWS and, when prompted, enter the 6-digit number from the AWS website on your phone keypad.
- If you don't receive a call from AWS, choose Sign in to sign in to the console again and start over. Or see Lost or unusable Multi-Factor Authentication (MFA) device to contact support for help.
-
After you verify your phone number, you can sign in to your account by choosing Sign in to the console.
-
The next step varies depending on the type of MFA you are using:
- For a virtual MFA device, remove the account from your device. Then go to the AWS Security Credentials page and delete the old MFA virtual device entity before you create> a new one.
- For a FIDO security key, go to the AWS Security Credentials page and deactivate the old FIDO security key before enabling a new one.
- For a hardware TOTP token, contact the third-party provider for help fixing or replacing the device. You can continue to sign in using alternative factors of authentication until you receive your new device. After you have the new hardware MFA device, go to the AWS Security Credentials page and delete the old MFA hardware device entity before you create a new one.
-
Confirm that IAM principals have appropriate access and permissions following the security event.
-
Address vulnerabilities and install the necessary patches using AWS SSM patch manager or any other 3rd party tool you use.
-
Replace any compromised/damaged files with clean versions from backup.
After you have successfully recovered from the security incident, you should conduct the exercises outlined in AWS Security Incident Response Guide, Post-Incident activity. This will help you document lessons learned from the incident, measure and improve the effectiveness of your incident response capabilities, and implement additional controls to prevent a similar incident from recurring.
In this playbook, we have described the initial steps to take when a compromised credential security event occurs within your AWS account(s). This includes engaging AWS support, detecting a compromise, analysis of events within your account(s), containing the security event, eradicating the threat, recovering your account(s) to a “known-good” operational state, and post-incident activities including lessons learned. As a next step, please review the following AWS resources, to help improve your Incident response capabilities:
CloudTrail Log Structure
When searching through CloudTrail logs, it is important to understand the various fields contained within a CloudTrail event record. For a full list, please refer to the official CloudTrail documentation.
Field | Description |
---|---|
eventTime | The date and time the request was completed, in coordinated universal time (UTC). |
eventName | The requested action, which is one of the actions in the API for that service |
eventSource | The service that the request was made to. This name is typically a short form of the service name without spaces plus .amazonaws.com. |
userIdentity | Information about the IAM identity that made a request. For more information, see CloudTrail userIdentity element. |
resources | A list of resources accessed in the event. The field can contain the following information:
|
awsRegion | The AWS Region that the request was made to, such as us-east-2 |
sourceIPAddress | The IP address that the request was made from. For actions that originate from the service console, the address reported is for the underlying customer resource, not the console web server. |
Events
There are a number of actions threat actors will perform after compromising account credentials. While it is impractical to list every possible action, here are some patterns to search for during your investigation.
Note: The API actions below do not necessarily indicate a security incident has occurred. Evaluate all logged activity within the context of your investigation.
Changes to SAML / OIDC identity provider configuration
Threat actors may create or modify SAML/OIDC identity provider configurations to avoid detection and maintain persistence within your cloud environment.
API Action | Description |
---|---|
iam:CreateSAMLProvider | Creates an IAM resource that describes an identity provider (IdP) that supports SAML 2.0 |
iam:DeleteSAMLProvider | Deletes a SAML provider resource in IAM. |
iam:UpdateSAMLProvider | Updates the metadata document for an existing SAML provider |
iam:CreateOpenIDConnectProvider | Creates an IAM entity to describe an identity provider (IdP) that supports OpenID Connect (OIDC). |
iam:AddClientIDToOpenIDConnectProvider | Adds a new client ID (also known as audience) to the list of client IDs already registered for the specified IAM OpenID Connect (OIDC) provider resource |
iam:DeleteOpenIDConnectProvider | Deletes an OpenID Connect identity provider (IdP) resource object in IAM |
iam:RemoveClientIDFromOpenIDConnectProvider | Removes the specified client ID (also known as audience) from the list of client IDs registered for the specified IAM OpenID Connect (OIDC) provider resource object |
iam:UpdateOpenIDConnectProviderThumbprint | Replaces the existing list of server certificate thumbprints associated with an OpenID Connect (OIDC) provider resource object with a new list of thumbprints |
Changes to IAM Configuration
Threat actors may create or modify IAM principals, credentials, or permissions to avoid detection or maintain persistence in your cloud environment.
API Action | Description |
---|---|
iam:ChangePassword | Changes the password of the IAM user who is calling this operation |
iam:CreateUser | Creates a new IAM user for your AWS account |
iam:CreateRole | Creates a new role for your AWS account |
iam:CreateGroup | Creates a new group |
iam:AttachUserPolicy | Attaches the specified managed policy to the specified user |
iam:AttachRolePolicy | Attaches the specified managed policy to the specified IAM role |
iam:AttachGroupPolicy | Attaches the specified managed policy to the specified IAM group |
iam:ListAccessKeys | Returns information about the access key IDs associated with the specified IAM user |
iam:CreatePolicyVersion | Creates a new version of the specified managed policy |
iam:UpdateLoginProfile | Changes the password for the specified IAM user |
iam:CreateAccessKey | Creates a new AWS secret access key and corresponding AWS access key ID for the specified user |
iam:UpdateAccessKey | Changes the status of the specified access key from Active to Inactive, or vice versa. |
iam:DeactivateMFADevice | Deactivates the specified MFA device and removes it from association with the username for which it was originally enabled |
Changes to Logging & Monitoring Configurations
Threat actors may disable resource monitoring or delete logs to avoid detection or deter incident investigations.
cloudtrail:DeleteTrail | Deletes a trail – disabling the delivery of CloudTrail events to an Amazon S3 bucket, CloudWatch Logs, or Amazon EventBridge |
---|---|
cloudtrail:StopLogging | Suspends the recording of AWS API calls and log file delivery for the specified trail |
cloudtrail:UpdateTrail | Updates the settings that specify delivery of log files |
cloudtrail:DeleteEventDataStore | Deletes a CloudTrail Lake Event Date Store |
GuardDuty:DeleteDetector | Deletes an Amazon GuardDuty detector and disables GuardDuty in a particular region |
GuardDuty:DeletePublishingDestination | Deletes the publishing destination, where findings are exported to |
accessanalyzer:DeleteAnalyzer | Deletes the specified analyzer, disabling the IAM access analyzer service for that region. |
config:DeleteConfigRule | Deletes the specified AWS Config rule and all of its evaluation results |
config:DeleteConfigurationRecorder | Deletes the AWS Config configuration recorder |
config:DeleteDeliveryChannel | Deletes the AWS Config delivery channel |
config:StopConfigurationRecorder | Stops recording configurations and configuration changes for the specified recording group |
S3 Bucket Configuration Changes
Threat actors may modify or delete S3 bucket configurations in order to exfiltrate data.
S3:ListBuckets | Returns a list of all buckets owned by the authenticated sender of the request |
---|---|
S3:CreateBucket | Creates a new bucket |
S3:DeleteBucketPublicAccessBlock | Removes the PublicAccessBlock configuration for an Amazon S3 bucket |
S3:PutBucketPolicy | Adds or updates a policy on a bucket |
S3:DeleteBucketPolicy | Deletes the policy from the bucket |
S3:PutObjectAcl | Sets the access control list (ACL) permissions for an object in an Amazon S3 bucket |
S3:DeleteObjectAcl | Delete the access control list (ACL) of an object |
S3:PutBucketCORS | Sets the CORS configuration for your bucket |
S3:DeleteBucketCORS | Deletes the CORS configuration information set for the bucket |
S3:PutBucketEncryption | Configure default encryption and Amazon S3 Bucket Keys for an existing bucket |
S3:DeleteBucketEncryption | Resets the default encryption for the bucket as server-side encryption with Amazon S3 managed keys (SSE-S3) |
Other Actions
Here are some other actions that you should highlight during your investigation
kms:DisableKey | Sets the state of a KMS key to disabled. This change temporarily prevents use of the KMS key for cryptographic operations. |
---|---|
kms:ScheduleKeyDeletion | Schedules the deletion of a KMS key. |
kms:PutKeyPolicy | Attaches a key policy to the specified KMS key. Key policies are the primary way to control access to KMS keys. |
kms:CreateKey | Creates a unique customer managed KMS key in your AWS account and Region. |
EC2:DescribeInstances | Describes EC2 instances within your account. |
EC2:RunInstances | Launches EC2 instances within your account. |
EC2:CreateVpc | Creates a VPC within your account. |
EC2:DescribeVpcs | Describes VPCs within your account. |
EC2:CreateSecurityGroup | Creates a security group. A security group acts as a virtual firewall for your instance to control inbound and outbound traffic. |
EC2:DeleteSecurityGroup | Deletes a security group. |
RDS:CreateDBInstance | Creates an RDS database instance. |
RDS:DeleteDBInstance | Deletes an RDS database instance. |