OpenTelemetry is the CNCF standard for observability providing simple code instrumentations and vendor neutrality while supporting logs, metrics, and traces. This increases the demand for observability backends supporting this format. Analyzing observability data is one of the main use cases for OpenSearch. The observability plugin enables powerful trace analytics amongst other things. Data Prepper closes the gap between OpenTelemetry-instrumented applications and OpenSearch by providing the required field mappings.
The first part of the session provides a general overview of Data Prepper. We will discuss some of Data Prepper’s key offerings to help you ingest observability data into OpenSearch.
In the second part, a productive use case within SAP will show how OpenTelemetry and OpenSearch come together. This is achieved by a managed Data Prepper component for ingestion and several predefined dashboards for analysis.
We will show the working solution from SAP Cloud Logging and explain our Data Prepper based architecture and deployment. This will include operational challenges with Data Prepper running on Kubernetes, such as memory configuration and stability. An important aspect is ingestion performance, for which we present our load test setup and learnings.