Key Takeaways

• Enhanced Network Security and Troubleshooting: VPC Flow Logs are invaluable for security analysis, troubleshooting connectivity issues, and detecting anomalies, enabling organizations to bolster their network security and efficiency.
• Compliance and Performance Monitoring: These logs are essential for compliance monitoring and performance assessment, helping organizations meet regulatory requirements and maintain optimal network performance.
• Cost Management and Resource Planning: VPC Flow Logs assist in detailed cost allocation and capacity planning, ensuring that resources are scaled appropriately based on accurate traffic analysis.

What is VPC Flow Logs?

VPC Flow Logs are a feature offered by cloud service providers like Amazon Web Services (AWS), Microsoft Azure and Google Cloud Platform (GCP). They capture detailed information about network traffic within a Virtual Private Cloud (VPC) or cloud network environment. These logs provide visibility into network activities, offering insights into the flow of data packets, including their source, destination, and the time of communication.

Introduction

Amazon Web Services (AWS) VPC Flow Logs is a feature designed to capture and provide information about the IP traffic that flows to and from network interfaces within your Virtual Private Cloud (VPC). This data can be published to various destinations, including AWS CloudWatch Logs, AWS S3, or AWS Kinesis Data Firehose. Flow logs serve several important purposes, such as diagnosing security group rule issues, monitoring incoming and outgoing traffic, and determining traffic directions.

In this blog, we will review common use cases and problems associated with storing and processing VPC Flow Logs. In Part 2 of this blog series, we look at ways to optimize VPC Flow Logs for cost and performance through an Observability pipeline, while maintaining their analytical significance.

Use Cases

• Security Analysis: Analyzing and Monitoring Security Groups and Network ACLs: VPC Flows allow you to analyze traffic patterns, identify source and destination IP addresses, and monitor the ports and protocols used. This information is valuable for assessing the effectiveness of your security groups and network access control lists (ACLs).‍
• Troubleshooting: Identifying and Diagnosing Connectivity Issues: When connectivity issues arise, VPC Flows provide detailed information about the traffic flow, helping you identify any bottlenecks, misconfigurations, or issues with routing. It aids in troubleshooting by providing insights into the source and destination of the traffic.‍
• Compliance Monitoring: Meeting Compliance Requirements (e.g., PCI DSS, HIPAA): Many compliance standards require organizations to monitor and log network traffic. VPC Flows help you meet these requirements by providing a comprehensive record of all traffic entering and leaving your VPC. This information can be used for auditing and compliance reporting.‍
• Performance Monitoring: Analyzing Network Performance: VPC Flow Logs include data on network latency, packet loss, and other performance metrics. By analyzing this information, you can gain insights into the performance of your network infrastructure and identify areas for optimization.‍
• Anomaly Detection: Detecting Unusual or Suspicious Network Activity: By regularly analyzing VPC Flow Logs, you can establish baseline patterns of normal behavior. Deviations from these patterns can be indicative of security incidents, such as unauthorized access or a potential compromise. Anomaly detection using flow logs enhances your overall security posture.‍
• Capacity Planning: Planning for Resource Scaling: VPC Flow Logs provide information on the volume of traffic between different components of your architecture. This data can be used for capacity planning, helping you anticipate increases in network traffic and plan for the scaling of resources as needed.‍
• Cost Allocation: Allocating Costs to Different Resources or Departments: Understanding the data transfer patterns between different resources within your VPC allows you to allocate costs accurately. This is particularly useful in scenarios where multiple departments or projects share the same AWS account.

‍Key Features and Benefits

• Non-Intrusive: Flow log data is collected without impacting network throughput or latency, ensuring that your network performance remains unaffected.
• Customizable: You can create flow logs for your entire VPC, specific subnets, or individual network interfaces, tailoring the monitoring to your needs.
• Flexible Destinations: Flow logs can be published to AWS CloudWatch Logs, AWS S3, or AWS Kinesis Data Firehose, making it easy to integrate with other AWS services for analysis and alerting.
• Resource Compatibility: You can create flow logs for network interfaces created by various AWS services, such as Elastic Load Balancing, AWS RDS, and more.

Creating Flow Logs

It's important to note that flow log data isn't real-time but has a slight delay before it's available for analysis. When creating a flow log, you need to specify:

• Resource: The resource you want to monitor (VPC, subnet, or network interface).
• Traffic Type: The type of traffic to capture (accepted, rejected, or all traffic).
• Destinations: The destinations for the flow log data (CloudWatch Logs, S3, or Kinesis Data Firehose).

Flow Log Record Fields

Flow log records are log events that describe network traffic flows within your VPC. They include various fields, each providing specific information about the traffic flow. For a complete list of VPC flow log record fields, see this list at AWS.

• Version: The version of VPC Flow Logs.
• Account ID: The AWS account ID of the network interface owner.
• Interface ID: The ID of the network interface.
• Source Address (srcaddr) and Destination Address (dstaddr): The IP addresses associated with the traffic.
• Source Port (srcport) and Destination Port (dstport): The port numbers involved in the traffic.
• Protocol: The IANA protocol number of the traffic.
• Packets: The number of packets transferred.
• Bytes: The number of bytes transferred.
• Start and End Times: Timestamps for the start and end of the traffic flow.
• Action: Indicates whether the traffic was accepted or rejected.
• Log Status: The status of flow log logging.
• VPC ID: The ID of the VPC.
• Subnet ID: The ID of the subnet.
• Instance ID: The ID of the associated instance.
• TCP Flags: Bitmask value for TCP flags.
• Type: The type of traffic (IPv4, IPv6, EFA).
• Packet-Level Source and Destination Addresses (pkt-srcaddr, pkt-dstaddr): Original source and destination IP addresses.
• Region: The AWS region.
• Availability Zone ID (az-id): ID of the Availability Zone.
• Sublocation Type and ID: Information about sublocations.
• Packet Source and Destination AWS Service: AWS service related to the source and destination IP addresses.
• Flow Direction: The direction of the flow.
• Traffic Path: The path taken by egress traffic.

Limitations of AWS VPC Flow Logs

• Flow logs can't be enabled for VPCs peered with your VPC unless the peer VPC is in your account: Enabling flow logs for VPCs peered with yours is restricted unless the peer VPC belongs to the same AWS account. This limitation may impact visibility in interconnected environments.
• Configuration changes for flow logs are limited, and you may need to create a new one to adjust settings: Adjusting configuration settings for flow logs is constrained, and significant changes may necessitate the creation of a new flow log. This limitation can impact the agility of log adjustments.
• Aggregation intervals may vary based on network interface type: The aggregation intervals for flow logs may vary depending on the type of network interface. This variability could impact the precision of the logged data and affect the granularity of analysis.
• Flow logs do not capture all types of IP traffic, including DNS traffic, Windows license activation traffic, and more: Not all types of IP traffic are captured by flow logs. This limitation is notable for scenarios where capturing specific types of traffic, such as DNS requests or Windows license activation traffic, is crucial for comprehensive monitoring and analysis.

Conclusion

VPC Flow Logs provide valuable insights into network traffic within your VPC. By creating and analyzing flow logs, you can enhance your network security, monitor traffic patterns, and better understand your VPC's behavior. Understanding the capabilities and limitations of flow logs is crucial for effective network management and troubleshooting. Read VPC Flow Logs Part 2 where we talk about how an Observability pipeline can be used for optimizing and enriching VPC Flow log data.  Check our blog page for more tips on optimizing telemetry data for DevOps and Security teams, and general best practices for observability and log management.

Learn More

For more information on how you can save 50% or more on your SIEM and observability costs with the AI-powered Observability Pipeline, Read the Observo.ai White paper, Elevating Observability with AI.