The Challenge

Saas Software is immensely popular because it allows customers to get the latest enhancements and feature upgrades faster without having to install updates or migrate to newer software versions. That’s why a Major SaaS Software Development company was so eager to improve their developer productivity to deliver software faster and more reliably.

The company relies on Datadog for security and observability analytics as well as application tracing and debugging. Like many organizations, the data ingested into Datadog was growing by 30-35% per year. This was causing several problems. The team faced several unexpected overage charges during periods of increased data volume or usage. Infrastructure costs were also becoming a challenge for them. Perhaps most concerning, it was becoming increasingly more difficult to isolate and debug issues. False alerts flooded Datadog and meaningful ones were diluted by too much noise in the data.  

Knowing the importance of constantly improving their software offerings, the VP of  IT Infrastructure tasked his team with finding solutions to control costs and provide their engineers with a better way to find and resolve issues. Initially, they looked at individual data types including Application logs, VPC Flow Logs, CloudTrail logs, and Database logs to see if they could ration their limited ingest budget to help control costs, but this had little impact. Also, they were concerned about missing something important in the data sources they were excluding.

The Solution

Ultimately, they knew a longer-term solution was needed. One of their team members watched an explainer video for Observo.ai and was intrigued by the ability to massively reduce data volumes without losing signal and the features that enable faster issue resolution.

The team chose Observo.ai for these key features:

• Anomaly Detection
• Data Optimization and Reduction
• Searchable, Low-cost Data Lake
• Compliance and Sensitive Data Discovery

With Observo.ai, they built a dynamic pipeline that uses AI to learn from the data flowing through it to constantly improve optimizations. The Observo.ai Smart Summarizer surfaces anomalies and distinguishes them from normal events to reduce alert fatigue. Observo.ai examines patterns in the data to learn which events should be more highly prioritized than others. The company used this to strengthen its debugging efforts.

Their Datadog license was locked in place for the next 2 years. Still, by lowering ingested data volume, they had an immediate impact on reduced infrastructure and stopped getting unanticipated overage fees during spike data periods. They also decided to retain data for shorter periods in Datadog to speed up queries and created a data lake with Observo.ai. If they ever need to analyze data older than a couple of weeks, they can easily use natural language queries to find it, use the Observo.ai rehydration capability, and send it back to Datadog or another tool. The low-cost Observo.ai data lake allowed them to retain data for much longer, improving their security posture.

As a SaaS software provider, their customers are in a wide range of industries where data security is subject to various regulations and standards. Observo.ai can automatically detect sensitive information even when it shows up in unexpected places. This gave them the confidence to tout data security more prominently as a feature of their products which enhanced customer trust.

Results

They initially reduced total data volume by more than 75% using the data source-specific algorithms from Observo.ai. Due to the continuous learning nature of Observo.ai machine learning technology, this will improve as more data is analyzed. They saved 45% on their total budget for Datadog by optimizing data retention with their new data lake and reducing the compute toll on running Datadog.

By applying sentiment analysis to events, their team was able to quickly prioritize which data could help them find and fix critical issues. This enabled them to shorten issue debugging times by 42%, which led to cutting more than 20% off their software delivery cycles. 

“Our SaaS customers are thrilled with our faster release schedules. With Observo.ai, we didn’t have to compromise on quality and still saved more than 45% on Datadog costs." 

- Sandeep L., VP IT Infrastructure