Overview

CloudTrail logs are a type of log generated by Amazon Web Services (AWS) as part of its CloudTrail service. AWS CloudTrail records API calls made within an AWS account, providing a history of activity including actions taken through the AWS Management Console, AWS Command Line Interface (CLI), and AWS SDKs. For example, CloudTrail events are generated for actions such as EC2 instances start/stop, S3 bucket read/write and IAM user creation/deletion. CloudTrail logs serve as the single source of truth for all activities that take place in an AWS account. Hence, this serves a vital role in security, governance and operational visibility. This is Part 1 in a series on CloudTrail Logs. Read Part 2 where we will discuss how Observo.ai can optimize CloudTrail Log data and route it to any SIEM or log analytics platform.

Use Cases

Security Monitoring and Threat Detection

1. CloudTrail logs provide a detailed audit trail of API activity within an AWS account, enabling security teams to monitor for suspicious behavior and potential security threats.
2. By analyzing CloudTrail logs, organizations can detect unauthorized access attempts, unusual patterns of activity, and potential security misconfigurations.
3. Security teams can set up alerts based on specific API actions or patterns of activity, allowing for real-time notification of potential security incidents.

Compliance and Governance

1. CloudTrail logs are essential for demonstrating compliance with various regulatory requirements and industry standards, such as PCI DSS, HIPAA, GDPR, and more.
2. Organizations can use CloudTrail logs to track and report on changes made to AWS resources, user activity, and access controls, providing evidence of adherence to compliance policies.
3. CloudTrail logs also support forensic investigations and audit trails, helping organizations maintain accountability and transparency in their AWS environments.

Operational Troubleshooting and Debugging

1. CloudTrail logs can assist in troubleshooting operational issues and diagnosing system errors within an AWS environment.
2. By analyzing CloudTrail logs, DevOps teams can identify the root cause of performance issues, configuration errors, and service disruptions.
3. CloudTrail logs provide visibility into API calls made by AWS services and applications, helping teams pinpoint issues and implement timely resolutions.

Change Management and Configuration Tracking

1. CloudTrail logs serve as a historical record of changes made to AWS resources and configurations over time.
   Organizations can use CloudTrail logs to track changes to IAM policies, security group rules, S3 bucket configurations, and other critical settings.
2. CloudTrail logs enable organizations to maintain a comprehensive change management process, ensuring accountability and transparency for infrastructure changes.

Auditing and Forensic Analysis

1. CloudTrail logs support auditing and forensic analysis of AWS activity, allowing organizations to investigate security incidents, data breaches, and compliance violations.
2. Security teams can analyze CloudTrail logs to reconstruct the timeline of events leading up to an incident, identify unauthorized actions, and assess the impact of security breaches.
3. CloudTrail logs provide valuable evidence for incident response, enabling organizations to take appropriate remediation actions and strengthen their security posture.

CloudTrail Log Deep Dive

{
  "eventVersion": "1.07",
  "userIdentity": {
     "type": "AssumedRole",
     "principalId": "AROAEXAMPLEID:ExampleRole",
     "arn": "arn:aws:sts::123456789012:assumed-role/ExampleRole/ExampleSessionName",
     "accountId": "123456789012",
     "accessKeyId": "AKIAEXAMPLEKEY",
     "sessionContext": {
        "sessionIssuer": {
           "type": "Role",
           "principalId": "AROAEXAMPLEID:ExampleRole",
           "arn": "arn:aws:iam::123456789012:role/ExampleRole",
           "accountId": "123456789012",
           "userName": "ExampleRole"
        },
        "webIdFederationData": {},
        "attributes": {
           "mfaAuthenticated": "false",
           "creationDate": "2023-04-06T12:34:56Z"
        }
     }
  },
  "eventTime": "2023-04-06T12:36:56Z",
  "eventSource": "s3.amazonaws.com",
  "eventName": "GetObject",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "203.0.113.123",
  "userAgent": "aws-cli/1.23.4 Python/3.9.7 Darwin/21.1.0 botocore/1.23.4",
  "requestParameters": {
     "bucketName": "example-bucket",
     "key": "example-object.txt",
     "x-amz-id-2": "EXAMPLE-ID-STRING",
     "x-amz-request-id": "EXAMPLE-REQUEST-ID"
  },
  "responseElements": {
     "x-amz-request-id": "EXAMPLE-REQUEST-ID",
     "x-amz-id-2": "EXAMPLE-ID-STRING",
     "ETag": "EXAMPLEETAG"
  },
  "requestID": "EXAMPLE-REQUEST-ID",
  "eventID": "EXAMPLE-EVENT-ID",
  "eventType": "AwsApiCall",
  "apiVersion": "2021-10-08",
  "recipientAccountId": "123456789012"
}

CloudTrail Event Contents

1. eventSource: Indicates the AWS service that generated the event, in this case, s3.amazonaws.com.
2. eventName: Specifies the action performed, which is GetObject indicating that an object was read from the S3 bucket.
3. awsRegion: The AWS region where the event occurred, in this case, us-east-1.
4. userIdentity: Provides information about the user or role that performed the action. In this example, it's an assumed role (type: AssumedRole) named ExampleRole.
5. requestParameters: Contains details about the request, including the bucket name (bucketName) and the key (object name) of the object that was read (key).
6. responseElements: Contains elements returned by the service in response to the request, such as ETag of the object read.
7. sourceIPAddress: IP address from which the request originated.
8. userAgent: Information about the client making the request, such as the AWS CLI version used.

Problems with CloudTrail logs

1. Computationally Expensive: As you can imagine, CloudTrail logs tend to be extremely voluminous making it hard to process. Managing the large volume of CloudTrail logs requires scalable infrastructure and can strain resources, potentially leading to performance issues.‍
2. Cost Management: Storing and processing CloudTrail logs can incur costs, necessitating careful management of log retention, storage options, and optimization of analysis workflows to control expenses.‍
3. Needle in the haystack: CloudTrail logs contain detailed information but parsing and interpreting them for actionable insights can be complex. From a security context, 99+% of events are not relevant. Identifying the anomalous logs ends up being a challenge for even the most sophisticated security teams.

Conclusion

CloudTrail logs are the definitive, central record of all actions occurring within an AWS account. This comprehensive data set encompasses every activity across the account, providing a detailed audit trail that is crucial for maintaining a secure, compliant, and well-managed cloud environment. These logs offer in-depth insights into user interactions, API calls, and other operational events, making them an essential tool for monitoring, diagnosing, and troubleshooting various cloud operations. By keeping track of all account activities, CloudTrail logs enable organizations to detect and respond promptly to potential security threats, ensure adherence to governance policies, and achieve operational transparency across all services and resources. In addition, they facilitate efficient incident response and root cause analysis, allowing organizations to maintain robust security postures and optimize their AWS environment.

Continue to learn about CloudTrail logs in Part 2 in this series where we talk about how an AI-Powered Observability Pipeline Like Observo.ai can be used for optimizing and enriching CloudTrail log data.  Check our blog page for more tips on optimizing telemetry data for DevOps and Security teams, and general best practices for observability and log management.

Learn More

For more information on how you can save 50% or more on your SIEM and observability costs while cutting the time to resolve critical incidents by more than 40% with the AI-powered Observability Pipeline, read the Observo.ai White paper, Elevating Observability with AI.