A Blue Team guide to AWS Cloudtrail monitoring
In this post, we stay in the cloud services monitoring and we’ll tackle AWS Cloudtrail. Just like Azure, Cloudtrail can be tricky and there’s a lot of room for improvement, but the documentation is much better than Azure’s. The logs are also streamed in JSON, therefore parsing and field extraction in a SIEM should be easy. The searches in this post were written for Splunk but it might be easy to recreate in a different SIEM.
Once again, just like Azure, this is not a complete guide, first because it would be time consuming but also because companies have different threat models and corporate policies. However, I believe it will help to get you started with some essential and basic searches to build on-top of.
Requirements
This depends on your SIEM. Most SIEM or log management platforms provide apps or connectors to easily ingest logs from AWS Cloudtrail. For Splunk specifically, you will likely need the following apps:
Splunk Add-on for Amazon Web Services: Responsible for the data ingestion
Splunk app for AWS :Responsible for interpreting the data and provides some out of the box dashboards
If you’re using ArcSight, you’ll find the connectors on the ArcSight Marketplace. If such app is not available for the solution you are using, you can script the pulling with the Cloudtrail API. The logs provided are in JSON format so the parsing should be easy.
AWS Management Console
AWS Management Console events logged by Cloudtrail under the signin.amazonaws.com event source and enables us to track and monitor user sign-in activity and identify potential security issues. Below are a few examples of how to identify issues with the events from the sign-in logs.
Successful or failed sign-in attempts
To know if a sign-in attempt was a success or a failure, we need to search for the ConsoleLogin events in the eventName field. It will display all sign-in activity done through the AWS Management Console. The userName field contains the user name that generated the event and the responseElements.ConsoleLogin contains Failure for a failed sign-in attempt and Success for a successful one. Some additional fields are available, such as additionalEventData.MFAUsed that will tell whether the sign-in attempt was done using MFA or not and userIdentity.type which specifies whether the user is a Root user or IAM user.
As example, the search below returns successful authentications without multi-factor authentication. It can help detect suspicious logins or accounts on which MFA is not enforced.
sourcetype="aws:cloudtrail" eventName="ConsoleLogin" "responseElements.ConsoleLogin"=Success "additionalEventData.MFAUsed"=No
Using these events we can monitor and detect multiple types of security issues or successful attacks, such as:
- brute-forcing attempts
- users logging in without MFA
- successful logins from previously unseen country
- successful logins from known malicious IP address
Failed authentications by source
This simple search returns a table of failed authentication, including the source IP, country, city and the reason why the authentication failed.
sourcetype="aws:cloudtrail" eventName="ConsoleLogin" "responseElements.ConsoleLogin"=Failure
| iplocation sourceIPAddress
| stats count by userName, userIdentity.accountId, eventSource, sourceIPAddress, Country, City, errorMessage
| sort - count
Using this search as a base, you can enrich it by excluding known good IPs and build a list of known attackers in a lookup table. Then, with the list of known attacker, you can create a similar search to look for successful authentication from known attacker’s IPs.
AWS Elastic Compute Cloud (EC2)
Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud. In this section we will cover multiple type of events to monitor and that can be used to detect suspicious activity. Cloudtrail provides detailed events for any action related to an EC2 instance, including some features such as VPCs, volumes, routing tables, network ACLs and so on. While not all events event are relevant from a security perspective, some can be useful to detect misconfiguration or abuse of large EC2 instances.
Abuse of GPU instances for Cryptocurrency Mining
Creation of GPU instances is uncommon in most businesses as they serve specific purposes like parallel computation for artificial intelligence and machine learning. It’s also a good use case for abuse. An employee with malicious intentions could spawn multiple GPU instance to mine cryptocurrencies. Fortunately, we can detect this kind of abuse by looking at the RunInstances actions. These events are generated when a new instance is powered on. The field requestParameters.instanceType contains the type of the instance. The GPU EC2 instances to look for are the following:
- p3.2xlarge
- p3.8xlarge
- p3.16xlarge
- p3dn.24xlarge
- p2.xlarge
- p2.8xlarge
- p2.16xlarge
- g3s.xlarge
- g3.4xlarge
- g3.8xlarge
- g3.16xlarge
Below is an example of Splunk search to identify GPU instances that have been started.
sourcetype="aws:cloudtrail" eventSource="ec2.amazonaws.com" eventName="RunInstances"
| spath output=instanceType path=requestParameters.instanceType
| spath output=minCount path=requestParameters.instancesSet{}.items{}.minCount
| search instanceType IN ("p3.2xlarge", "p3.8xlarge", "p3.16xlarge", "p3dn.24xlarge", "p2.xlarge", "p2.8xlarge", "p2.16xlarge", "g3s.xlarge", "g3.4xlarge", "g3.8xlarge", "g3.16xlarge")
| stats count by eventSource, eventName, awsRegion, userName, userIdentity.accountId, sourceIPAddress, userIdentity.type, requestParameters.instanceType, responseElements.instancesSet.items{}.instanceId, responseElements.instancesSet.items{}.networkInterfaceSet.items{}.privateIpAddress, minCount
| fields - count
The results contain a variety of useful information that can help quickly identify if the activity was legitimate.
- One or multiple GPU instance created from a source IP address that is known to be malicious or from or from a previously unseen country might indicate a compromised AWS account.
- One or multiple GPU instances created from an IAMUser user type instead of an AssumedRole.
- EC2 API throttling hit by a single user in a short amount of time.
- The number of instances started in a single API call (field minCount)
Security Groups
Security groups act as virtual firewall for EC2 instances to control inbound and outbound traffic. By searching the AuthorizeSecurityGroupIngress action in the eventName field we can start looking for dangerous network configurations. Below is a description of the relevant fields for this type of event.
| Field | Description |
|---|---|
| requestParameters.groupId | Security group ID |
| requestParameters.ipPermissions.items{}.fromPort | From port (range) |
| requestParameters.ipPermissions.items{}.ipProtocol | Protocol |
| requestParameters.ipPermissions.items{}.ipRanges.items{}.cidrIp | IPv4 network range to allow or deny (CIDR notation) |
| requestParameters.ipPermissions.items{}.toPort | To port (range) |
| requestParameters.number | Rule number |
| sourceIPAddress | Source IP of user creating the ACL |
In the following example, we look for rules allowing inbound traffic on port 22 from any IPs. Then we look for the associated instance IDs and append them to the list.
sourcetype="aws:cloudtrail" eventSource="ec2.amazonaws.com" eventName="AuthorizeSecurityGroupIngress"
| spath output=fromPort path=requestParameters.ipPermissions.items{}.fromPort
| spath output=toPort path=requestParameters.ipPermissions.items{}.toPort
| spath output=cidrIp path=requestParameters.ipPermissions.items{}.ipRanges.items{}.cidrIp
| spath output=groupId path=requestParameters.groupId
| spath output=accountId path=userIdentity.accountId
| spath output=type path=userIdentity.type
| search fromPort=22 toPort=22 AND cidrIp="0.0.0.0/0"
| spath output=ipPermissions path=requestParameters.ipPermissions.items{}
| mvexpand ipPermissions
| fields - fromPort, toPort, cidrIp
| spath input=ipPermissions
| spath output=cidrIp path=ipRanges.items{}.cidrIp input=ipPermissions
| join groupId
[ search index=aws eventName=RunInstances earliest=-7d
| fields "responseElements.instancesSet.items{}.groupSet.items{}.groupId", "responseElements.instancesSet.items{}.instanceId"
| rename responseElements.instancesSet.items{}.groupSet.items{}.groupId as groupId, "responseElements.instancesSet.items{}.instanceId" as instanceId]
| stats values(instanceId) by groupId, userName, accountId, type, sourceIPAddress, cidrIp, fromPort, toPort, ipProtocol
The table returned by this search not only gives us the misconfigured security groups, but also the affected instance IDs and account IDs (took me a while before figuring out how to write this one). If you wish to extract the public IP addresses for all instances IDs and append them to the results, you will need a subscription to AWS Config or use the DescribeInstances API endpoint.
Network ACLs
The CreateNetworkAclEntry action allows us to monitor newly created ACL entries (rules) to detect dangerous network configurations. Network ACLs are optional and provide an additional layer of security in addition to security groups.
What are the differences between Network ACLs and Security Groups? Network ACLs are applicable at the subnet level, so any instance in the subnet with an associated ACL will follow the rules of the ACL. That’s not the case with security groups as they have to be assigned explicitly to one or multiple instances.
If a security group allows traffic from ANY to port 22, before reporting this (depending on your company’s policy), I suggest confirming the following:
- Is there an ACL applied to the subnet concerned?
- If there is, does it restrict SSH to a whitelist of IPs?
Inversely, you can also investigate a Network ACL that allows SSH from ANY.
- Are the instances on this subnet protected by a security group?
- If they are, does it restrict SSH to a whitelist of IPs?
Below is a description of the relevant fields for this type of event.
| Field | Description |
|---|---|
| requestParameters.aclProtocol | Protocol number |
| requestParameters.cidrBlock | IPv4 network range to allow or deny (CIDR notation) |
| requestParameters.egress | True if outbound, False if inbound |
| requestParameters.number | Rule number |
| requestParameters.portRange.from | From port (range) |
| requestParameters.portRange.to | To port (range) |
| requestParameters.ruleAction | Allow or Deny |
| sourceIPAddress | Source IP of user creating the ACL |
In the following example, we search for creation of Network ACL rules allowing inbound connections from any sources.
sourcetype="aws:cloudtrail" eventSource="ec2.amazonaws.com" eventName=CreateNetworkAclEntry
| spath output=cidrBlock path=requestParameters.cidrBlock
| spath output=ruleAction path=requestParameters.ruleAction
| search cidrBlock=0.0.0.0/0 ruleAction=Allow
AWS Simple Storage Service (S3)
Detecting public S3 buckets
Public S3 buckets has been a big issue since the creation of the service. If a bucket is set to public by mistake, it can leak confidential and sensitive information. More importantly, if a bucket is public and also has the write permissions on it, its content could be altered by a malicious attacker. By looking for the PutBucketAcl event name where the grantee URI is AllUsers we can identify and report the open buckets. The search below does exactly that:
sourcetype=aws:cloudtrail AllUsers eventName=PutBucketAcl errorCode=Success
| spath output=userIdentityArn path=userIdentity.arn
| spath output=bucketName path=requestParameters.bucketName
| spath output=aclControlList path=requestParameters.AccessControlPolicy.AccessControlList
| spath input=aclControlList output=grantee path=Grant{}
| mvexpand grantee
| spath input=grantee
| search Grantee.URI=*AllUsers
| rename userIdentityArn as user
| table _time, src,awsRegion Permission, Grantee.URI, bucketName, user
Conclusion
I hope this was helpful in getting you started if you’ve just set up the ingestion of Cloudtrail and wondered where to start. If you have questions or would like to exchange ideas, you can email me at [email protected] or by DM on Twitter.
I will also be at DEFCON so message me if you think I could help you with something or simply to hang out or say hi. I’ll be pretty busy but I can always try to find some time to meet new people and help out.