Security operations teams are drowning in data. The rapid increase in security events, logs, and observability metrics makes it increasingly difficult to detect threats effectively. Data volume growth leads to high storage and processing costs, inefficient threat detection, and difficulty in extracting actionable insights from noisy datasets. To solve these challenges, Observo AI has partnered with AWS Security Lake to help organizations optimize, route, and store security data in a structured and cost-effective manner.

As an official AWS Security Lake partner, Observo AI empowers security teams to:

• Optimize data before ingestion to ensure only the most valuable security data is processed.
• Transform data from a wide range of sources into the Open Cybersecurity Schema Framework (OCSF)
• Route data to the right tools for real-time security analytics and SIEM optimization.
• Store enriched and normalized security data in AWS Security Lake for long-term retention.
• Retrieve and analyze archived data on demand for threat hunting, forensic investigations, and compliance audits
• Route data from AWS Security Data Lake to other SIEM, Log Management or Observability tools

Data Optimization: Reducing Noise and Cost While Retaining Valuable Insights

One of the key benefits of integrating Observo AI with AWS Security Lake is the ability to optimize security data before ingestion. Most security tools today ingest raw, high-volume logs that are often noisy, redundant, and costly to store. Observo AI intelligently processes these logs to:

• Filter unnecessary data: Remove duplicate events, excessive telemetry, and low-value logs that do not contribute to security insights.
• Summarize key details: Instead of storing millions of logs that have identical information, Observo AI extracts key details to retain only relevant security signals.
• Enrich data with context: Adding metadata and correlating logs across different sources enhances detection accuracy and investigation speed.
• Normalize to OCSF format: Observo AI automatically translates security data into OCSF, ensuring compatibility with AWS Security Lake and other analytics tools.

By optimizing security data before ingestion, Observo AI helps organizations reduce storage and compute costs associated with traditional SIEMs and cloud analytics platforms, without sacrificing visibility.

Routing Security Data to the Best Tools

A major challenge for security teams is getting the right data to the right tools in real-time. Not all security logs need to be sent to a SIEM, and not all analytics tools require the same level of detail. Observo AI enables:

• Dynamic data routing based on priority and need: Critical security events can be sent to SIEMs or SOAR platforms for immediate action, while lower-priority logs can be stored in AWS Security Lake for later analysis.
• Multi-destination routing: Security data can be simultaneously routed to different destinations, such as AWS Security Lake for cost-effective storage, a SIEM for real-time analysis, and a threat intelligence platform for correlation.
• Format transformation: Logs are automatically converted into the optimal format for each tool, ensuring seamless integration across security ecosystems.
• For organizations considering migrating from one SIEM to AWS SL - Observo AI can route to both while the migration is being completed. 

This intelligent routing approach reduces SIEM ingestion costs while ensuring security teams always have access to the right data at the right time.

AWS Security Lake as a Cost-Effective Security Data Archive

While real-time security data is crucial for immediate detection and response, long-term storage of enriched security data is essential for forensic investigations and compliance. AWS Security Lake provides a scalable and cost-effective solution for storing security data, and Observo AI enhances this capability by ensuring only relevant, structured, and enriched security data is archived.

Key benefits of using AWS Security Lake with Observo AI include:

• Efficient data retention: Store optimized security logs in Amazon S3 using AWS Security Lake's OCSF-based schema, ensuring a structured and searchable archive.
• Scalability and cost savings: Reduce reliance on expensive SIEM storage by offloading logs to AWS Security Lake while keeping them easily retrievable when needed.
• Compliance and audit readiness: Maintain long-term security data archives for compliance with industry regulations such as SOC 2, GDPR, and HIPAA.

What is OCSF? 

The Open Cybersecurity Schema Framework (OCSF) is an open-source initiative designed to standardize security data across different tools and platforms. It was created to address the challenges of working with fragmented security data, which often comes in incompatible formats, making correlation and analysis difficult. OCSF provides a common schema that normalizes security event data, enabling easier integration, automation, and threat detection across multiple security solutions.

The framework was initiated by a coalition of cybersecurity and technology leaders, including AWS, Splunk, and IBM, to establish a shared language for security data. It allows vendors and organizations to structure their logs and telemetry in a way that is both extensible and interoperable, ensuring that security teams can quickly and accurately analyze threats. By using OCSF, businesses can reduce the complexity of security operations, improve visibility across their environments, and accelerate incident response.

Observo AI transforms data from a wide range of sources into the OCSF format, ensuring seamless integration with AWS Security Lake and other security platforms.

How Observo AI Optimizes and Transforms Data for AWS Security Lake

As an AWS Security Lake source partner, Observo AI ensures that security data is efficiently optimized, transformed, and delivered in the OCSF format required by AWS. One example of this process is how we handle VPC Flow Logs, a common but often overwhelming data source. Without proper optimization, raw VPC Flow Logs can generate excessive noise, leading to unnecessary costs and storage inefficiencies.

To address this, Observo AI applies a VPC Optimizer, which parses VPC Flow Logs and filters out redundant or low-value data while preserving the critical network traffic insights needed for security investigations. This process significantly reduces data volume before further transformation, making it easier to extract meaningful security signals. Once optimized, the data is standardized into our internal Observo format, a structured intermediary step that ensures consistency across multiple data types.

After optimization and standardization, the data is passed through our OCSF serializer, which dynamically maps it into the OCSF schema used by AWS Security Lake. Data is routed to AWS Security Lake which is a specialized AWS S3 storage location.  The data is stored in Parquet so that it works optimally with AWS services such as Athena for efficient querying and analysis. This structured approach not only optimizes security data ingestion but also enhances searchability and correlation across diverse security sources.

Another key benefit of using Observo AI is our ability to support OCSF version updates seamlessly. OCSF is an evolving standard, and keeping pace with its changes can be challenging for security teams. Our serializer is continuously updated to reflect the latest schema versions, ensuring ongoing compatibility with AWS Security Lake. 

By integrating data optimization, flexible routing, and OCSF standardization, Observo AI streamlines security operations, reduces data management overhead, and enhances threat detection capabilities. Whether it's VPC Flow Logs or other security telemetry, Observo AI ensures that organizations can extract the most value from their security data while keeping costs and complexity under control, and seamlessly integrating with AWS services.

On-Demand Retrieval for Future Investigations

Storing security data in AWS Security Lake is only part of the solution—security teams must also be able to quickly retrieve and analyze archived data when needed. Observo AI provides seamless access to stored data, enabling teams to:

• Run retrospective investigations: Query AWS Security Lake for historical security events to uncover attack patterns and identify past threats.
• Correlate past incidents with current threats: Enriched logs stored in AWS can be cross-referenced with new threat intelligence to detect long-term attack campaigns.
• Enhance threat hunting and analytics: Retrieve specific datasets from AWS Security Lake and feed them into advanced analytics tools for deeper security insights.

By retaining optimized security data in AWS Security Lake and enabling on-demand retrieval, Observo AI helps security teams conduct forensic investigations without the high costs associated with SIEM storage.

Conclusion: A Smarter Approach to Security Data Management

With Observo AI and AWS Security Lake, organizations can take control of their security data, reducing costs while improving security effectiveness. Observo AI ensures that only the most relevant, enriched, and structured data is ingested, routed to the best tools, and stored efficiently for future investigations.

By leveraging this partnership, security teams can: 

• Reduce SIEM and cloud storage costs with intelligent data parsing and optimization. 
• Enhance threat detection and response by routing data to the right tools. 
• Store enriched security logs efficiently for compliance and future investigations. 
• Retrieve data on-demand for retrospective analysis and threat hunting.
• Maintain currency with changes to the OCSF format

Observo AI’s security data pipeline, combined with AWS Security Lake’s scalable storage and analytics capabilities, is a game-changer for security teams looking to optimize their data strategy.