Overview

As a pivotal element within your networking configuration, logs generated by Network Firewalls hold immense importance from both security and compliance standpoints. These logs serve as a source of valuable information, encompassing records of network traffic details like source and destination IP addresses, ports, protocols, timestamps, and the actions (e.g., allowed or denied) taken by the firewall for each connection or packet.

With Firewall log volumes growing by 25-35% annually, the expenses associated with processing and storing these logs have become a substantial concern for numerous organizations. In part 1 of this series, we took a look at common use cases and problems associated with storing and processing Firewall Logs. In this blog post, we'll delve into how Observo.ai addresses this challenge by optimizing Firewall Logs for cost efficiency and enhanced performance, all while preserving their analytical value. Stay tuned for part 3 of this series, where we'll examine a customer case study.

Optimizing Firewall Logs

1. Aggregate repetitive log events

The sporadic and intermittent nature of communication between network clients and servers results in a corresponding bursty pattern reflected in Firewall logs. Observo addresses this by leveraging its robustly scalable reduction engine, which adeptly consolidates these intermittent bursts of logs that occur within condensed time frames. Consider an example where a firewall receives network traffic from 2 unique sources. 

Without Observo.ai in the pipeline, we see highly repetitive and voluminous logs generated by the Firewall:

Timestamp                      Source IP             Destination IP     Port    Action       Rule        Bytes

2024-01-25 12:00:00     192.168.1.101        203.0.113.42       80      ALLOW      Rule1       1200

2024-01-25 12:00:05     192.168.75.222    203.0.113.42       80      ALLOW      Rule2       500

2024-01-25 12:00:10     192.168.1.101         203.0.113.43       80      ALLOW     Rule1        3500

2024-01-25 12:00:15     192.168.75.222     203.0.113.43       80      ALLOW     Rule2        800

2024-01-25 12:00:20     192.168.1.101         203.0.113.42       80      ALLOW     Rule1        2000

2024-01-25 12:00:25     192.168.75.222     203.0.113.42       80      ALLOW     Rule2       1300

2024-01-25 12:00:30     192.168.1.101         203.0.113.43       80      ALLOW     Rule1        550

2024-01-25 12:00:30     70.18.1.101             203.0.113.43       80      DENY        Rule1        75

2024-01-25 12:00:30     71.17.1.99              203.0.113.43       80      DENY        Rule1        120

2024-01-25 12:00:35     192.168.75.222     203.0.113.43       80      ALLOW     Rule2        3200

2024-01-25 12:00:40     192.168.1.101         203.0.113.42       80      ALLOW     Rule1         750

2024-01-25 12:00:45     192.168.75.222     203.0.113.42       80      ALLOW     Rule2        1800

Using Observo.ai's reduction function, we can aggregate related traffic logs, making it simpler for people to understand network traffic patterns while significantly reducing the amount of data. Notice that repetitive traffic between the same source/destination pair is aggregated together. Traffic that is denied by the firewall is passed through. This is how our traffic data looks like after being processed by Observo.ai:

Timestamp                      Timestamp_end             Source IP           Destination IP     Port     Action       Rule       Bytes

2024-01-25 12:00:00     2024-01-25 12:00:40   192.168.1.101      203.0.113.42        80       ALLOW     Rule1      8000

2024-01-25 12:00:05     2024-01-25 12:00:05   192.168.75.222  203.0.113.42        80       ALLOW     Rule2     7600

2024-01-25 12:00:30     2024-01-25 12:00:30   70.18.1.101           203.0.113.43       80       DENY        Rule1      75

2024-01-25 12:00:30     2024-01-25 12:00:30   71.17.1.99             203.0.113.43       80       DENY        Rule1      120

‍

Through this process, the aggregation of Firewall logs routinely achieves a substantial reduction in log volumes, often surpassing 60%. This reduction in log clutter not only optimizes storage but also significantly aids downstream Security Information and Event Management (SIM) systems. By aggregating these logs, Observo.ai effectively condenses and centralizes contextual details dispersed across numerous events into more concise and coherent entries. This consolidation process streamlines the information flow, empowering SIM systems to more efficiently process and analyze the data, ultimately enhancing their performance in detecting and responding to potential security threats within the network infrastructure.

2. Sample logs from Trust Zones

Firewall logs within a trusted zone are often considered less critical to analyze because the trusted zone typically consists of internal networks or systems where communication is presumed to be secure and trustworthy. Some firewalls like Palo Alto’s Next Gen firewall even annotate network traffic logs with information about trusted zones. As these zones contain well-known and authorized traffic, there's an assumption that the risk of malicious activities or unauthorized access is relatively lower compared to external or less trusted areas. Therefore, logs within trusted zones may receive less scrutiny or priority in analysis, focusing more on monitoring and analyzing logs from untrusted or external zones where potential threats or security breaches are more likely to originate.

Observo.ai's adaptable policy engine offers a range of options to tailor sampling and filtering strategies according to specific needs. This flexibility empowers users to define and implement customized rules for handling traffic logs originating within the organization's trusted zones. By employing these customizable strategies, Observo.ai allows for the efficient sampling and reduction of redundant or less critical data within these trusted zones, thereby optimizing the transmission of high-value, meaningful information to downstream SIM systems. This selective forwarding of high-signal data helps streamline the workload of SIMs, ensuring that they receive and process only the most relevant and valuable information, enhancing overall system performance and security monitoring capabilities.

3. Route different log categories to different destinations

Firewall logs consist of numerous categories. Typically, the most commonly analyzed categories are:

1. Security Threats: Logs documenting attempted breaches, unauthorized access, malware activities, and successful or thwarted security threats targeting the network's integrity and confidentiality.
2. Traffic Logs: Records detailing network traffic patterns, such as incoming/outgoing connections, accessed services, source IPs, and used protocols, crucial for monitoring and analyzing network traffic.
3. System Errors: Logs indicating technical issues within the firewall or network infrastructure, encompassing hardware failures, software glitches, and configuration errors impacting network operations.

Within many organizations, distinct categories of Firewall logs undergo processing and storage via various log destinations. For instance, an organization might direct Security Threat logs to a SIEM system while opting for a more cost-effective log destination for Traffic logs. Observo.ai's adaptable Routing engine facilitates the segregation and routing of diverse Firewall log categories toward their pertinent destinations. This capability empowers your organization to efficiently allocate logs to their designated destinations, ensuring a streamlined and targeted approach to log management, analysis, and storage.

4. Data lake

Numerous compliance frameworks, including PCI DSS, HIPAA, and ISO 27001, often stipulate the necessity for organizations to retain Firewall logs for extended durations, frequently exceeding a year. This prolonged retention period serves the purpose of ensuring comprehensive auditability, facilitating security investigations, and complying with regulatory standards. Observo.ai significantly streamlines the establishment of a Data Lake by offering a seamless process for storing and managing vast amounts of log data. Through Observo, organizations gain the capability to efficiently write data to cost-effective object storage solutions while leveraging the platform's functionality to store this data in a highly compressed Parquet format. This approach not only optimizes storage costs but also enhances data accessibility, query performance, and analytics capabilities, enabling organizations to meet stringent compliance requirements while efficiently managing and utilizing their log data for security and operational insights.

Conclusion

Firewall logs are crucial for monitoring network security events in enterprises, but they often contain voluminous data with minimal analytical value. This log noise inflates analytics systems, strains budgets, and hinders the identification of genuine security threats. Observo.ai can dramatically reduce the volume of firewall logs, summarize and aggregate repetitive logs, dynamically sample for more data reduction, route to multiple destinations and create a security data lake for retention and compliance. 

Keep an eye out for part 3 of this series, where we delve into a customer use case, providing practical insights into Observo's effectiveness in real-world scenarios.

Learn More

For more information on how you can save 50% or more on your SIEM and observability costs with the AI-powered Observability Pipeline, Read the Observo.ai White paper, Elevating Observability with AI.‍