Configuring log aggregation and observability for sonataflow

GitHub Actions · jmagak · commit 66af45044a6e · 2026-04-15T12:25:40.000+02:00
diff --git a/assemblies/extend_orchestrator-in-rhdh/assembly-configure-opentelemetry-for-sonataflow-workflows.adoc b/assemblies/extend_orchestrator-in-rhdh/assembly-configure-opentelemetry-for-sonataflow-workflows.adoc
@@ -17,7 +17,7 @@ include::../modules/extend_orchestrator-in-rhdh/proc-enable-opentelemetry-for-so
 include::../modules/extend_orchestrator-in-rhdh/proc-configure-telemetry-exporters.adoc[leveloffset=+1]
 
 // SonataFlow OpenTelemetry attributes and events
-include::../modules/extend_orchestrator-in-rhdh/proc-configure-observability-tools.adoc[leveloffset=+1]
+include::../modules/extend_orchestrator-in-rhdh/ref-observability-configuration-examples.adoc[leveloffset=+1]
 
 // Troubleshooting OpenTelemetry integration
 include::../modules/extend_orchestrator-in-rhdh/ref-troubleshoot-opentelemetry-connectivity.adoc[leveloffset=+1]
diff --git a/modules/extend_orchestrator-in-rhdh/proc-configure-telemetry-exporters.adoc b/modules/extend_orchestrator-in-rhdh/proc-configure-telemetry-exporters.adoc
@@ -1,24 +1,25 @@
 :_mod-docs-content-type: PROCEDURE
 
 [id="configure-telemetry-exporters_{context}"]
-= Export telemetry data to an observability platform
+= Configure telemetry data exporters for observability platforms
 
 [role="_abstract"]
-When monitoring serverless workflows in a distributed environment, export trace data efficiently to an external observability platform (such as Jaeger or an OpenTelemetry Collector). Configuring your export strategy and externalizing environment variables makes sure that your telemetry data is delivered reliably without hardcoding sensitive configurations in your production builds.
+To monitor serverless workflows in a distributed environment, export trace data to an external observability platform, such as Jaeger or an OpenTelemetry Collector. By configuring an export strategy and externalizing environment variables, you ensure reliable telemetry delivery and avoid hardcoding configurations in production builds.
 
 .Prerequisites
 
 * You have enabled OpenTelemetry in your workflow.
+* An observability platform (Jaeger or OpenTelemetry Collector) is available in your cluster.
 
 .Procedure
 
-. Configure your export strategy
+. Define your export strategy.
 +
-Choose one of the following export strategies based on the requirements of your observability platform:
+Choose an export strategy that matches your observability platform requirements:
 
-.. OTLP exporter with batch processing (Recommended)
+** OTLP exporter with batch processing (Recommended)
 +
-For production environments, using an OTLP exporter with batch processing reduces network overhead and improves performance, as shown:
+For production environments, use an OTLP exporter with batch processing to reduce network overhead and improve performance.
 +
 [source,bash]
 ----
@@ -34,9 +35,9 @@ quarkus.otel.bsp.export.timeout=2s
 quarkus.otel.bsp.max.queue.size=2048
 ----
 
-.. Direct Export to an external platform
+** Direct export to an external platform
 +
-For simpler setups or direct integrations, you can configure a direct export as shown:
+For development or simple integrations, use a direct export configuration.
 +
 [source,bash]
 ----
@@ -46,9 +47,9 @@ quarkus.otel.exporter.otlp.protocol=grpc
 quarkus.otel.traces.exporter=cdi
 ----
 
-. Externalize configuration for production deployments
+. Externalize the configuration for production deployments.
 +
-To maintain secure and flexible production deployments, use environment variables to externalize your OpenTelemetry configuration:
+Use environment variables to externalize your OpenTelemetry configuration. This ensures that your deployment remains secure and flexible across environments.
 +
 [source,bash]
 ----
diff --git a/modules/extend_orchestrator-in-rhdh/proc-enable-opentelemetry-for-sonataflow-workflows.adoc b/modules/extend_orchestrator-in-rhdh/proc-enable-opentelemetry-for-sonataflow-workflows.adoc
@@ -6,28 +6,27 @@
 [role="_abstract"]
 Enable OpenTelemetry in your SonataFlow project to begin collecting distributed traces, metrics, and logs.
 
-The OpenTelemetry integration for SonataFlow provides the following:
+To enable observability features such as tracing and metrics in the SonataFlow runtime, you must add the OpenTelemetry addon and configure the workflow properties. The `sonataflow-addons-quarkus-opentelemetry` addon provides a standard configuration with minimal setup required.
 
-Distributed tracing:: Tracks workflow execution across multiple services and steps.
-Metrics collection:: Monitors performance, duration, and success rates.
-Log aggregation:: Centralized logging with trace correlation.
-Context propagation:: Maintains trace context across workflow boundaries and async operations.
+The OpenTelemetry integration for SonataFlow includes the following capabilities:
 
-[NOTE]
-====
-To enable observability features like tracing and metrics in the SonataFlow runtime, use the `sonataflow-addons-quarkus-opentelemetry` addon. This addon provides a standard configuration that requires little additional setup.
-====
+* Distributed tracing: Track workflow execution across services and steps.
+
+* Metrics collection: Monitor performance, duration, and success rates.
+
+* Log aggregation: Centralize logs with trace correlation.
+
+* Context propagation: Maintain trace context across workflow boundaries and asynchronous operations.
 
 .Prerequisites
 
-* You have installed and configured a SonataFlow Operator.
-* You have access to deploy observability infrastructure.
-* You have Kubernetes or OpenShift cluster with appropriate resources.
-* You understand OpenTelemetry concepts.
+* You have installed and configured the SonataFlow Operator.
+* You have `cluster-admin` or equivalent permissions to deploy observability infrastructure and modify ConfigMaps.
+* A Kubernetes or OpenShift cluster is available.
 
 .Procedure
 
-. Add the SonataFlow OpenTelemetry addon to your `QUARKUS_EXTENSIONS` environment variable during the image build process:
+. Add the OpenTelemetry addon to the `QUARKUS_EXTENSIONS` environment variable during the image build process:
 +
 [source,bash]
 ----
@@ -36,7 +35,7 @@ export QUARKUS_EXTENSIONS="${QUARKUS_EXTENSIONS},org.apache.kie.sonataflow:sonat
 
 . Open the `{workflow-name}-props` ConfigMap for your workflow.
 
-. In the `application.properties` section, enable the OpenTelemetry integration:
+. In the `application.properties` section, enable the OpenTelemetry integration and configure the service attributes:
 +
 [source,bash]
 ----
@@ -69,37 +68,27 @@ sonataflow.otel.spans.enabled=true
 sonataflow.otel.events.enabled=true
 ----
 
-. Save the ConfigMap and restart the workflow pod.
+. Save the ConfigMap and restart the workflow pod to apply the changes.
 
 .Verification
 
-* Check OpenTelemetry addon is loaded:
+* Verify that the OpenTelemetry addon is loaded by checking the pod logs:
 +
 [source,bash]
 ----
 kubectl logs -n workflows deployment/onboarding-workflow | grep "sonataflow-addons-quarkus-opentelemetry"
 ----
 
-* Verify trace export:
+* Verify successful trace export:
 +
 [source,bash]
 ----
-# Look for successful trace exports
 kubectl logs -n workflows deployment/greeting | grep -i "export\|batch"
 ----
 
-* Check Jaeger health:
+* Confirm that the observability backend (for example, Jaeger) is receiving data:
 +
 [source,bash]
 ----
-# Verify Jaeger is receiving data
 kubectl logs -n observability deployment/jaeger | grep -i "span\|trace"
-----
-
-* Test with debug exporter:
-+
-[source,bash]
-----
-# Temporarily enable console logging
-%dev.quarkus.otel.traces.exporter=logging
 ----
diff --git a/modules/extend_orchestrator-in-rhdh/ref-observability-configuration-examples.adoc b/modules/extend_orchestrator-in-rhdh/ref-observability-configuration-examples.adoc
@@ -1,18 +1,16 @@
 :_mod-docs-content-type: CONCEPT
 
-[id="configure-observability-tools_{context}"]
-= Tools for configuring observability
+[id="observability-configuration-examples_{context}"]
+= Observability tool configuration examples
 
 [role="_abstract"]
-The following are working examples for two essential observability tools that integrate seamlessly with SonataFlow OpenTelemetry implementation.
+Use the following examples to deploy and integrate Jaeger and Loki with the SonataFlow OpenTelemetry implementation. These examples include configurations for development and production environments.
 
-. Jaeger distributed tracing tool.
+. Jaeger distributed tracing
 +
 Jaeger provides distributed tracing visualization for SonataFlow workflows.
 +
-Deploy Jaeger on OpenShift or Kubernetes as shown in the following examples:
-+
-All-in-one deployment (development or testing):::
+.Jaeger all-in-one deployment (development and testing)
 +
 [source,yaml]
 ----
@@ -104,8 +102,8 @@ spec:
     targetPort: 16686
   type: ClusterIP
 ----
-
-OpenShift route for UI access:::
++
+.OpenShift route for UI access
 +
 [source,yaml]
 ----
@@ -124,8 +122,10 @@ spec:
     termination: edge
     insecureEdgeTerminationPolicy: Redirect
 ----
-
-Workflow Configuration for Jaeger:::
++
+.Workflow configuration for Jaeger
++
+Add these properties to the `application.properties` file of your workflow:
 +
 [source,bash]
 ----
@@ -137,8 +137,8 @@ quarkus.otel.traces.exporter=cdi
 # Additional Jaeger-specific propagation
 quarkus.otel.propagators=tracecontext,baggage,jaeger
 ----
-
-Production Deployment with Elasticsearch:::
++
+.Production deployment with Elasticsearch
 +
 For production environments, use the Jaeger Operator with Elasticsearch storage:
 +
@@ -176,13 +176,11 @@ spec:
         memory: 512Mi
 ----
 
-. Loki log aggregation tool.
-+
-Grafana Loki provides scalable log aggregation with native OpenTelemetry support, enabling comprehensive log analysis and correlation with traces. Loki natively supports OpenTelemetry Protocol (OTLP) for direct log ingestion from SonataFlow workflows.
+. Loki log aggregation
 +
-Deploy Loki on OpenShift or Kubernetes as shown in the following examples:
+Loki supports OpenTelemetry Protocol (OTLP) for direct log ingestion from SonataFlow workflows.
 +
-Loki configuration for OpenTelemetry:::
+.Loki configuration for OpenTelemetry
 +
 [source,yaml]
 ----
@@ -237,8 +235,8 @@ data:
             prefix: index_
             period: 24h
 ----
-
-Loki deployment:::
++
+.Loki deployment
 +
 [source,yaml]
 ----
@@ -321,10 +319,10 @@ spec:
     targetPort: 9096
   type: ClusterIP
 ----
-
-. To configure workflow for Loki:
 +
-* Direct Connection to Loki (Recommended for simplicity):
+.Workflow configuration for Loki and Jaeger
++
+To route logs to Loki and traces to Jaeger, use the following configuration:
 +
 [source,bash]
 ----
@@ -353,27 +351,11 @@ quarkus.otel.resource.attributes=\
   deployment.environment=production
 ----
 
-. Optional: OpenTelemetry collector for advanced processing:
-+
-For production environments requiring log processing, enrichment, or multi-backend export, you can optionally deploy an OpenTelemetry collector between your workflows and the observability backends.
-+
-The following are the benefits of using a collector:
-
-* Log enrichment with Kubernetes metadata
-* Filtering and sampling
-* Multi-destination export (for example, logs to both Loki and external SIEM)
-* Centralized processing and transformation
-
-. To quickly setup the collector:
+. Optional: OpenTelemetry collector for advanced processing
 +
-[source,bash]
-----
-# Change workflow configuration to send to collector instead
-quarkus.otel.exporter.otlp.endpoint=http://otel-collector.observability.svc.cluster.local:4317
-----
-
-. To configure collector:
+Deploy an OpenTelemetry collector between workflows and backends for advanced log processing, filtering, and multi-destination export.
 +
+.Collector pipeline configuration
 [source,yaml,subs="+attributes,+quotes"]
 ----
 # Collector routes to both Jaeger and Loki
diff --git a/modules/extend_orchestrator-in-rhdh/ref-troubleshoot-opentelemetry-connectivity.adoc b/modules/extend_orchestrator-in-rhdh/ref-troubleshoot-opentelemetry-connectivity.adoc
@@ -4,70 +4,69 @@
 = Troubleshoot OpenTelemetry connectivity
 
 [role="_abstract"]
-Diagnose and resolve missing observability data and context propagation.
+Diagnose and resolve issues with missing observability data or failed context propagation in SonataFlow.
 
-If traces or logs are missing, you lose the visibility required to debug application issues or trace execution paths across services. You must quickly diagnose and resolve misconfigurations in your OpenTelemetry deployment so that monitoring is restored.
+.OpenTelemetry troubleshooting guide
+[cols="25%,35%,40%", options="header"]
+|===
+| Symptom
+| Potential cause
+| Resolution
 
-Traces do not appear::
+| Traces do not appear in the dashboard.
+| OpenTelemetry is disabled or the endpoint is unreachable.
+| Verify the `quarkus.otel.enabled` property and test endpoint connectivity.
 
-.. Check workflow configuration:
-+
-[source,bash]
-----
-# Verify OpenTelemetry is enabled
-kubectl get cm onboarding-workflow-props -n workflows -o yaml
+| Authentication errors (`401`/`403`).
+| Missing or invalid authorization headers.
+| Configure the `quarkus.otel.exporter.otlp.headers` property with a valid token.
 
-# Check pod logs for OpenTelemetry initialization
-kubectl logs -n workflows deployment/onboarding-workflow | grep -i "otel\|trace"
-----
+| High memory usage in the collector.
+| Large telemetry batches or high traffic volume.
+| Implement a `memory_limiter` processor in the collector configuration.
 
-.. Verify connectivity:
+| Context is lost between workflow steps.
+| Incorrect propagator configuration.
+| Ensure `quarkus.otel.propagators` includes all required formats (for example, `tracecontext` and `baggage`).
+|===
+
+.Diagnosing missing traces
+
+.. Verify that OpenTelemetry is enabled in the workflow ConfigMap:
 +
 [source,bash]
 ----
-# Test Jaeger endpoint from workflow pod
-kubectl exec -n workflows deployment/greeting -- curl -v http://jaeger-collector.observability.svc.cluster.local:4317
+kubectl get cm {workflow-name}-props -n workflows -o yaml
 ----
 
-Authentication issues::
-+
-.. For platforms requiring authentication, configure headers:
+.. Check the pod logs for initialization errors:
 +
 [source,bash]
 ----
-quarkus.otel.exporter.otlp.headers=authorization=Bearer ${API_TOKEN}
+kubectl logs deployment/{deployment-name} -n workflows | grep -i "otel"
 ----
 
-.. To authenticate the test:
+.. Test the connection to the Jaeger collector from within the workflow pod:
 +
 [source,bash]
 ----
-curl -v -X POST \
-  -H "Authorization: Bearer YOUR_TOKEN" \
-  -H "Content-Type: application/x-protobuf" \
-  https://your-endpoint.com/v1/traces
+kubectl exec deployment/{deployment-name} -n workflows -- curl -v http://jaeger-collector.observability.svc.cluster.local:4317
 ----
 
-High memory usage::
-+
-.. Configure memory limits in collector:
-+
-[source,yaml]
+.Configuring authentication headers
+If your observability platform requires authentication, add the following property to your `application.properties` file:
+
+[source,bash]
 ----
-processors:
-  memory_limiter:
-    check_interval: 1s
-    limit_mib: 1000
-    spike_limit_mib: 200
+quarkus.otel.exporter.otlp.headers=authorization=Bearer ${API_TOKEN}
 ----
 
-Context lost between steps::
-+
-Make sure there is a proper propagation:
-+
+.Resolving context propagation issues
+To ensure trace IDs are maintained across service boundaries, configure the following propagators and enable JSON logging to verify the IDs in the output:
+
 [source,bash]
 ----
-# Include all required propagators
+# Include required propagators
 quarkus.otel.propagators=tracecontext,baggage,jaeger
 
 # Enable JSON logging to verify trace IDs
