The Challenge

A Global 1000 E-commerce company struggled with the rapid growth in telemetry data that their security team analyzes with Splunk, Grafana, and other Observability tools in the cloud. Specifically, the increase in VPC Flow log and Firewall log volumes caused a spike in Splunk infrastructure costs on certain data sets and triggered daily indexing limit overage fees. As this deluge of data began piling up in block storage within their Splunk index, the team saw corresponding spikes in storage, compute, and egress costs. Additionally, their Splunk search performance slowed considerably as the index cycled through more and more data. 

Their security team was tasked with curbing the growth of their infrastructure bill and trying to get into compliance with their daily limits. Manual efforts to randomly sample security data helped reduce the volume but ultimately surfaced blind spots in their security posture as they were never sure if the sampled data reflected all of the actionable insights in the full dataset.

The Solution

The company began searching for an observability pipeline to help them manage the growth in log data that had accelerated over the past few years. They ultimately chose Observo due to its ability to optimize log data and because the drag-and-drop interface would allow them to deploy a pipeline within minutes and not a months-long integration.

Using Observo, they first created a full-fidelity data lake in AWS S3 where they sent all of their raw data. This data was stored in Parquet format and was easily searchable with Observo’s natural language queries. Parquet is highly compressible, allowing teams to store more data at a lower cost. On top of that, storing data in S3 is typically 1-3% of the cost of data stored in block storage within a SIEM index.

Next, they create data Pipelines in Observo to process & reduce VPC Flow logs and Firewall logs. These highly scalable Pipelines use algorithms specific to each data type that minimize useless data within each log and summarize similar logs for much more strategic sampling. These Pipelines optimized log volume by more than 80%. 

Results

“Our VPC flow log infrastructure costs spiked over a million dollars within a few months leading to a ton of anxiety. Observo was instrumental in controlling these costs." 
James T., Director III, Security Engineering

By retaiing less than 20% of their original VPC Flow log and Firewall data in a Splunk index, they were able to stay well within their budget and reduced total spend (including storage, egress, and compute costs) by more than 50%. Despite the drastic reduction in data stored in the index long-term, they were able to analyze all of the signal within their log data and plugged up any holes in their security analysis.

Their new data lake provided an easy way to retain data at a fraction of the cost. Finding and analyzing data with S3 with natural language queries made audits much simpler.

‍

Learn More

For more information on how you can save 50% or more on your security and observability infrastructure costs with the AI-powered Observability Pipeline, Read the Observo AI White paper, Elevating Observability with AI.