Introduction 

Fortinet FortiGate firewalls are crucial network security devices that help manage and protect your network by monitoring and controlling incoming and outgoing traffic. They do this based on a set of predetermined security rules. The logs generated by FortiGate firewalls are rich with information about network activities and security events, making them indispensable for both security and DevOps teams in enterprises.

For security teams, FortiGate firewall logs provide detailed insights into potential threats, unusual activities, and attempted breaches. This information is vital for identifying, investigating, and responding to security incidents quickly. Regular analysis of these logs can help in detecting patterns of malicious behavior, ensuring compliance with security policies, and enhancing the overall security posture of the organization.

For DevOps teams, these logs are equally important. They offer a clear view of network performance, helping to identify and troubleshoot issues that could affect the reliability and efficiency of applications and services. By monitoring traffic patterns and system behavior, DevOps teams can optimize resource allocation, improve application performance, and ensure smooth operations.

In this blog, we'll delve into common use cases for FortiGate firewall logs and discuss the challenges associated with storing and processing them. This will help both security and DevOps teams understand how to better leverage these logs to enhance security and operational efficiency. In Part 2 of the series, we will show how an AI-powered telemetry data pipeline like Observo AI can help solve these problems to leverage the value of this critical data.

Key Use Cases for FortiGate Firewall Logs

1. Security Monitoring: FortiGate logs are crucial for detecting and responding to security threats, providing real-time visibility into potential attacks and vulnerabilities.
2. Network Performance Monitoring: Traffic logs help optimize network performance by identifying bandwidth issues, congestion points, and inefficient routing.
3. Compliance and Auditing: Logs support compliance with industry regulations and auditing requirements by providing a record of network activities and security events.
4. Incident Response: In case of a security incident, logs can be analyzed to determine the scope and impact of the breach, aiding in response and recovery.
5. Policy Verification: Logs help ensure that FortiGate firewall policies are working as intended and that the network complies with security policies.
6. Trend Analysis: Over time, logs can identify trends and patterns in network traffic and security events, enabling proactive measures.
7. Resource Allocation: Traffic logs inform resource allocation decisions, ensuring effective distribution of network resources.

FortiGate Firewall Log Categories

FortiGate firewalls generate several types of logs, which can be broadly categorized as follows:

1. Traffic Logs: Record information about network traffic passing through the FortiGate unit.This log shows details of a traffic session, including source and destination IP addresses, ports, protocol, policy ID, and traffic statistics. This category tends to dominate log volume. 

date=2007-07-19 time=22:59:58 devname=xxxxx device_id=FGTxxxxx log_id=xxxxx type=traffic subtype=allowed pri=notice vd=root SN=xxxxx duration=130 user=N/A group=N/A policyid=1 proto=6 service=8080/tcp app_type=N/A status=accept src=xxx.xxx.xxx.xxx srcname=xxxxxx dst=xxx.xxx.xxx.xxx dstname=xxx.xxx.xxx.xxx src_int=xxxxx dst_int=xxxxx sent=299 rcvd=1759 sent_pkt=7 rcvd_pkt=6 src_port=56297 dst_port=8080 vpn=N/A tran_ip=0.0.0.0 tran_port=0 dir_disp=org tran_disp=noop‍

2. Security Logs (UTM Logs): Capture information about security threats and UTM (Unified Threat Management) events.
3. Event Logs: Contain information about FortiGate operations, administrative changes, system status, and hardware-related events. For example, this event log shows a system message about an expired license.

date=2007-07-23 time=05:30:23 devname=XXXX device_id=FGTxxxxx log_id=xxxxx type=event subtype=system pri=critical vd=root msg="FortiGuard Web Filter license is expired"‍

4. DNS Logs: Record DNS activity on FortiGate devices.
5. ZTNA Logs: A subtype of traffic logs specific to Zero Trust Network Access.

How FortiGate Logs are Ingested and Processed

1. FortiGate logs are typically sent to a Log Management System or Security Information and Event Management (SIEM) solution.
2. FortiGate devices allow administrators to configure log forwarding endpoints for different log types.
3. Common tools for processing FortiGate logs include FortiAnalyzer, Splunk, Elastic Stack (ELK), and other third-party SIEM solutions.
4. Once logs reach the designated endpoint, administrators set up dashboards and alerts to monitor network health and security.

Challenges in Storing and Processing FortiGate Logs

1. High Data Volume: Log volume scales with network traffic, leading to increased storage and processing costs as organizations grow.
2. Processing Complexity: Large volumes of logs make it computationally expensive to detect security threats in real-time.
3. Log Relevance: Only a subset of logs may be relevant for specific analysis or security purposes.
4. High Granularity: Traffic logs are generated for every network call, resulting in an explosion of events and expensive search and processing operations.
5. Cost Implications: The high volume and granularity of logs lead to increased costs for storage, processing, and analysis.

Conclusion

Stay tuned for Part 2 of this series, we'll explore strategies to optimize FortiGate log data, reducing noise and processing costs while maintaining effective security monitoring and analysis capabilities.

‍