How SAP analyzes OpenTelemetry signals using Data Prepper
2024-05-07 , Moskau

Besides providing an overview of Data Prepper, this session will focus on how an observability solution from SAP utilizes Data Prepper and OpenSearch to enable the ingestion and analysis of OpenTelemetry data. Included are operational challenges we faced as well as our performance test results.


OpenTelemetry is the CNCF standard for observability providing simple code instrumentations and vendor neutrality while supporting logs, metrics, and traces. This increases the demand for observability backends supporting this format. Analyzing observability data is one of the main use cases for OpenSearch. The observability plugin enables powerful trace analytics amongst other things. Data Prepper closes the gap between OpenTelemetry-instrumented applications and OpenSearch by providing the required field mappings.
The first part of the session provides a general overview of Data Prepper. We will discuss some of Data Prepper’s key offerings to help you ingest observability data into OpenSearch.
In the second part, a productive use case within SAP will show how OpenTelemetry and OpenSearch come together. This is achieved by a managed Data Prepper component for ingestion and several predefined dashboards for analysis.
We will show the working solution from SAP Cloud Logging and explain our Data Prepper based architecture and deployment. This will include operational challenges with Data Prepper running on Kubernetes, such as memory configuration and stability. An important aspect is ingestion performance, for which we present our load test setup and learnings.

David is a senior software engineer working on observability in OpenSearch at Amazon Web Services. He is a maintainer on the Data Prepper project. Prior to working at Amazon, he was the CTO at Allogy Interactive - a start-up creating mobile-learning solutions for healthcare.

This speaker also appears in:

Jannik is a software developer at SAP. He works on observability services of the SAP Business Technology Platform with a special focus on OpenTelemetry. He holds a Master of Science in computer science from the University of Heidelberg.