Introduction

Observability Pipelines have become vital tools for DevOps and Security teams to manage, control, store, route, and optimize telemetry data analyzed by Security Information and Event Management (SIEM), Application Performance Monitoring (APM), and Log management platforms. These teams spend hours every week trying to fit an increasingly large volume of data into the same size box. The manual efforts deployed to meet the challenge of this telemetry data, which our customers tell us is growing at the rate of 30-35% a year, include random sampling, rudimentary tools to reduce log volumes, or deploying static, rules-based pipelines. These pipelines require a deep level of expertise in both the schemas of the various data sources they’d like to analyze as well as the tools they use to analyze them. With a great deal of effort, they may be able to achieve a 20-25% reduction in log volume, but their static nature means that they need constant tuning to keep pace with the ever-evolving nature of data and the underlying changes in threats to their infrastructure.

An AI-powered Observability Pipeline like Observo.ai adds machine learning and artificial intelligence to elevate these pipelines to become always on, ever learning and improving, and automated to provide value within minutes of applying them. They employ powerful algorithm-based transforms to uniquely optimize each telemetry data type, route them to the right tools, and dynamically improve security, relevant alerting, cost control, and compliance.

This blog will cover the 6 most important benefits of an AI-powered Observability Pipeline. 

1. Data Optimization and Reduction - reduce data volume by 80% or more

Observo.ai can help you automate the optimization and reduction of security events and observability data right out of the box. Most telemetry data types, especially log data, contain more than 80% noise - data with zero analytical value. This can include duplicate fields, null values, legal disclaimers, header values, and other data that threaten your daily ingest limits and force you to make difficult decisions between data you can afford to analyze and data you hope doesn’t have the information you need to secure and optimize your environment.

Observo.ai uses data type-specific algorithms to reduce the volume of log data including Cloud Flow Logs, Firewall, OS, CDN, Application logs and others. In addition to reducing the size of individual logs, Observo.ai also uses the Smart Summarizer feature to dynamically sample out repetitive logs for further volume reduction. We use AI-based “patterns of patterns” summarization to find logs that are very similar to each other and only send a subset of logs through to analyze. Our customers can typically optimize their data stream by more than 80%, and our AI models are always learning from changes in your data to improve over time. 

By separating the noise from the signal, you’ll never pay to analyze telemetry data with no analytical value. This allows you to onboard new classes of data for a much more comprehensive picture of security and observability.

2. Smart Routing - collect data once, transform it, and route it to any set of destinations

Smart routing helps you transform any data from any source and route it to any destination in the right formats. You should never feel locked in to any vendor. Observo.ai allows you to reuse the data you have so evaluating or adding a new tool doesn’t require you to deploy agents and collectors on thousands of endpoints. By transforming existing data types, you only have to collect data once and route where it has the most value, including multiple tools and storage locations. 

3. Searchable, Low-cost Data Lake - keep more data, longer, for a lot less money

Observo.ai helps you create a full-fidelity data lake in low-cost cloud storage. This helps you comply with various requirements for retaining data, while dramatically lowering the cost of storing security event and observability data. Our most disciplined customers take a  snapshot of full-fidelity data. They transform it into Parquet format, which is highly compressible and makes it searchable with natural language queries. You don’t need to learn a specialized query language to find data in your lake – just describe your search request and our Large Language Model will convert it into the properly formatted query request. Because Parquet is so compressible and object cloud storage options like AWS S3 are relatively cheap, you can store data in a lake for about 1% of the cost of retaining it in block storage attached to your SIEM or log management tool.

Since most queries in analytics tools look for data from the previous 48 hours, you can be much more strategic about dropping data from your analytics platform - some of our more aggressive customers drop all data after a week. Because we can “rehydrate” any data from the data lake, transform it, optimize it, and re-route it back to whichever tool you choose, this offers tremendous cost savings without the fear of missing data. Retaining all of that data in your analytics tools can be very expensive and also require much more CPU to query a bloated data set. Many of our customers spend as much or more on CPU and storage as they do on the license itself. This best practice allows you to keep more data, for much longer periods, all for a dramatic reduction in infrastructure costs.

4. Compliance and Sensitive Data Discovery - detect and protect PII wherever it exists

Observo.ai can also help you protect sensitive data and stay in compliance with various regulatory requirements. Personally Identifiable Information (PII) requires special handling and care to adhere to privacy regulations like GDPR, CCPA, PCI, and others. A breach of this data not only puts you out of compliance but also risks the trust your customers have for your company.

An AI-powered Observability Pipeline goes beyond other pipelines that allow you to mask designated fields (we do that too). Observo.ai proactively detects and masks PII wherever it’s found in the stream. Open text fields, voice-to-text, and other innovations give your teams more data to analyze but also may expose PII in unexpected places. Machine learning enables the detection of this data whether it’s in a field marked Credit Card number or matches the format of a CC#, social security, or other PII anywhere else. We can obfuscate or hash it and work with your existing tools. 

5. Anomaly Detection - prioritize alerts and improve MTTR by more than 40%

Your teams are likely inundated with false positives and experience alert fatigue having to manually sort through events that are run-of-the-mill like resetting passwords and trying to find more impactful alerts that may indicate real security or stability threats. The Observo.ai pipeline learns what is normal for any given telemetry data type and alert. Our Sentiment Engine identifies anomalies and can integrate with common alert/ticketing systems like ServiceNow, PagerDuty, and Jira for real-time alerting. We can also add sentiment scores to logs to help your teams prioritize which needs attention now, which can wait, and which can be disregarded altogether. This removes alert fatigue and helps your teams resolve critical incidents by more than 40% by focusing on the biggest issues first.

6. Data Enrichment - add context to data for faster queries

Observo.ai can enrich telemetry data in the stream to add more context and better help you route and analyze it. Assign “sentiment” based on pattern recognition, or add 3rd party data like Geo-IP and threat intel to provide deeper context. This can help you fine-tune and speed up queries and also reduce the CPU toil of your analytics platform.

Conclusion

An AI-powered Observability Pipeline like Observo.ai is a huge leap forward in the practices of security analysis and observability. AI/ML models are always learning from your data as it evolves and can uncover savings beyond what can be achieved with traditional tools and manual workarounds. They help you break free from static, rules-based pipelines that need constant upkeep by very knowledgeable teams. Observo.ai reduces the volume of log data by more than 80% and helps you create a searchable, low-cost data lake allowing you to cut your total SIEM and Observability costs by over 50%. We use AI/ML to detect anomalies and prioritize alerts so your team can identify and resolve incidents more than 40% faster. We help you keep your customers’ trust by protecting sensitive data even when it shows up in unexpected places. 

Let us know what your biggest telemetry data challenges are. Chances are we can help you solve them.

Learn More

For more information on how you can save 50% or more on your SIEM and observability costs with the AI-powered Observability Pipeline, Read the Observo.ai White paper, Elevating Observability with AI.