Introduction

On April 24, 2024, Francis Odum, released his research report titled, “The Evolution of the Modern Security Data Platform” in The Software Analyst Newsletter. This report examines the evolution of modern security operations, tracing its evolution from a reactive approach to a proactive approach. It highlights the shift towards automation, threat intelligence integration, and controlling the costs of ingesting and storing data as crucial elements in enhancing cyber defense strategies. We were excited to see Observo.ai included as a key player in the emerging landscape for modern Security Operation Centers.

In this article, we will highlight some of the findings of this report and share how Observo.ai is addressing some of the biggest trends in security with our AI-Powered Telemetry Pipeline. 

Source: Francis Odum "The Evolution of the Modern Security Data Platform”

Executive Summary: The Evolution of the Modern Security Data Platform

This comprehensive research report delves into the dynamic evolution of security analysis, tracing its trajectory from conventional methods to contemporary paradigms. It explores the transition from reactive measures to proactive strategies, driven by the burgeoning complexity of digital threats and technological ecosystems.

Key Sections

Introduction to Security Analysis Evolution: A historical overview of security analysis, highlighting its origins in reactive practices like antivirus software and firewalls. It sets the stage for understanding the need for modernization in response to evolving cyber threats.

Emergence of Modern Techniques: Explores the rise of advanced methodologies such as threat intelligence, machine learning, and behavioral analytics, showcasing their pivotal role in proactive threat detection and mitigation. This section discusses how these techniques augment traditional security measures.

Challenges of the Digital Landscape: Examines the challenges posed by the expanding digital landscape, including the proliferation of connected devices, cloud computing, and the Internet of Things (IoT). It underscores the need for adaptable and scalable security solutions.

Collaborative Paradigm: Emphasizes the importance of collaborative efforts among security analysts, developers, and stakeholders. It illustrates how cross-functional teamwork enhances the implementation of robust security measures and fosters a culture of vigilance within organizations.

Continuous Adaptation in Security Practices: Stresses the necessity for security analysts to continuously adapt their strategies and tools in response to evolving threats. It advocates for staying abreast of emerging technologies and threat vectors, alongside investing in ongoing training and skill development.

Future Perspectives: Envisions forthcoming advancements in security analysis driven by artificial intelligence, automation, and decentralized technologies. It also cautions against the challenges posed by increasingly sophisticated adversaries and regulatory landscapes.

In conclusion, the report underscores the imperative for security analysts to evolve alongside the ever-changing threat landscape, advocating for the adoption of modern techniques and collaborative partnerships to effectively safeguard digital assets in today's dynamic cybersecurity milieu.

Observo.ai has worked with several customers who are implementing a modern SOC platform like the one described in this report. They have experienced stronger security, reduced manual workarounds, and have significantly controlled costs using this approach.

Explosion in Data Volume

“Legacy SIEM costs are largely indexed to data volume - meaning, the more stuff you ingest and index, the more you linearly pay. It’s now common knowledge that enterprises are accumulating data at record-setting speeds, meaning that SIEM costs are unfortunately also growing proportionally. In response, IT and security leaders have spent much of the last few years finding clever methods and tools to pre-process, reduce, and prioritize the data that they feed into these expensive systems.”
Francis Odum, “The Evolution of the Modern Security Data Platform”

Observo.ai was created to combat this meteoric rise in data volume. In fact, the idea for the company came when our founders were faced with the escalating costs in their SIEM renewal contract. When the proposal from their SIEM vendor came back in eight figures, something for which they could not get the budget approved,  they knew they had to come up with a better solution.

The idea for Observo.ai was conceived to help organizations control the growth of this data without losing any of the important signals contained within it. In a study of enterprise log and security event data, our team concluded that as much as 80% of log and security event data has zero analytical value. Sending all of that unusable data to your SIEM is a budget killer.

Observo.ai uses AI models to optimize this data in the stream before it hits the SIEM index and starts racking up numbers against your daily ingest limits. We can reduce the volume sent to your SIEM by 80% or more by summarizing normal events and separating out redundant or low-value data.

Alert Fatigue

“The primary problem has been the cost of ingesting and storing data on these platforms. Secondly, the rising volume of alerts generated from these solutions.”
Francis Odum, “The Evolution of the Modern Security Data Platform”

Not all alerts are created equally - but they can all clog up your security team’s inbox leaving them to wonder which alerts need attention now and which can be addressed later. Observo.ai uses machine learning to understand what is normal for each data type. The Observo.ai Sentiment Engine identifies anomalies and can assign sentiment values to events. By enriching events in the stream with positive or negative sentiment values, teams can better prioritize which alerts must be dealt with immediately. This helps teams identify and resolve critical incidents 40% faster. Helping your security teams be more productive and focus on the most meaningful alerts is all part of the modern SOC.

SIEM Vendor Lock-in

“In general, this SIEM vendor lock-in intensifies data management issues, it creates a lack of correlation among siloed sources, and necessitates data rehydration for investigations.”  
Francis Odum

Legacy SIEM vendors are incentivized to be the single destination for security data. The more data ingested into their index, the more they can charge their customers. But modern security teams are trying to balance sharply rising data volumes and the corresponding increase in SIEM licenses and infrastructure with flat to only modestly increasing budgets. Ripping and replacing is very difficult - installing agents and collectors across thousands of endpoints, applications, databases and firewalls could take months of time and take away your team from managing daily security tasks. It’s only when the prospect of massive increases in license costs and fees for daily ingest overages become so high that security teams would actually consider a switch.

Observo.ai gives you a much simpler way to balance the challenges of increasing  data volumes against flat budgets. With Observo.ai, you can route security data to multiple tools, and you don’t need to recollect data in order to do so. Observo.ai takes security data in the format you have and can transform it to any schema and route it to the tools you want in the right formats. This helps you route the most important data to more expensive tools and choose less expensive tools, including new SIEMs, for other classes of data. Having multiple SIEMs doesn’t mean that you need to collect the entire data sets multiple times - by transforming the data you have, you can collect once, optimize it, store a full-fidelity copy in low-cost data lake (see below), and route relevant sections to whatever tools make the most sense. Route data where it has the most value. 

Because we also reduce 80% or more of the data volume, this means you don’t have to choose between analyzing only the bare minimum and all of the data that gives insights into your security stance. This flexibility allows you to onboard new data types that may have been considered too expensive to analyze in your legacy SIEM including notoriously verbose sources like Firewall Logs and VPC Flow Logs.

In-House Cloud DIY Data Lake

“Security operations teams will increasingly adopt security data lakes without needing to replace existing SIEM solutions, allowing for better cost management and scalability.”
Francis Odum

The vast majority of SIEM queries are performed on data generated within the last two days. Still, many organizations keep months of data in their SIEM index. This can be a huge drag on performance, and rack up large storage costs. A better practice is to create a Security Data Lake for longer-term retention. Observo.ai makes it easy to create a data lake in low-cost cloud storage like AWS S3, Azure Blob, or GCP that is fully searchable with natural language queries. We store data in highly-compressible Parquet format to further control costs. Data can be stored in an Observo.ai data lake for about 1% of the cost of storing it in the SIEM index.

Observo.ai can rehydrate (send in the telemetry stream) data from the lake on-demand, transform and optimize it, and re-route to any SIEM tool in the right format for further analysis. Because of the ability to perform natural language queries on data stored in the lake, you don’t need a team of data scientists and engineers to pull the right data for an investigation. By separating the system of analysis (your SIEM) from your system of retention (Observo.ai data lake), you can reduce the total cost of operating a SIEM by 50% or more and retain data for much longer timeframes. 

Rise of Data ETL

“Companies like Observo have come in as data storage and management intermediaries. They act as an intelligent policy layer, absorbing filtering, and cleansing data (logs and events) before routing them into these large SIEMs. These players integrate with various apps, data management, and storage systems by intelligently filtering and managing data flow. This reduces unnecessary data replication, and managing data storage costs.”
Francis Odum

As we have discussed, the rise in security data brings the risk of a corresponding rise in total SIEM costs. Security teams are being tasked with keeping their spending within tight budgets. Without tools like Observo.ai, these teams are left with mundane, manual workarounds to try to harness the value of security data. Some of these include random sampling, excluding whole classes of data, or turning off data when volumes approach daily ingest limits. All of these are time-consuming and labor-intensive and introduce blindspots into your security mission. 

Observo.ai summarizes and samples data based on AI-based analysis in the stream. This helps ensure all of the data that matters gets into the best tools for analysis. We can automate this process to free up your teams to address security incidents instead of spending time worrying about ingest overage fees. 

Observo.ai can also route data to multiple tools. Many companies are trying to wean off legacy SIEM tools or at a minimum control the growth of data ingested into them. Observo.ai gives our customers the choice to send different classes of data to a different tool and to route anomalous data to more expensive tools and more normal data to lower cost tools or to an Observo.ai Data Lake. This is a huge protection against vendor lock-in and helps teams pick their optimal mix of tools and storage options without being held hostage by incumbent vendors or budget concerns.

Conclusion

“The Evolution of the Modern Security Data Platform” raises a lot of very interesting trends and best practices for security teams to consider. Observo.ai is a key part of implementing several of these recommendations. Observo.ai is the AI-Powered Pipeline for security data. To learn more about how Observo.ai can help you achieve a more modern approach to security, schedule a demo with us. You can also read our white paper, titled “Elevating Observability: Intelligent AI-Powered Pipelines.”