Custom ObservationHandler beans need to be registered after infrastructure handlers

[ttddyy]

I faced an ordering issue for registering custom ObservationHandler beans to the observation registry.

The use case is that I want to add a logic that adds tags to all spans. In order to do it, I wrote a custom ObservationHandler implementation that adds key-values to an observation context in onStop(). Then, I’m relying on DefaultTracingObservationHandler to convert those key-values in observation to span tags.

When I define my custom handler bean, ObservationHandlerGrouping is responsible for registering such handler beans to the registry in order.

spring-boot/spring-boot-project/spring-boot-actuator-autoconfigure/src/main/java/org/springframework/boot/actuate/autoconfigure/observation/ObservationHandlerGrouping.java

Lines 47 to 64 in 7e364cd

	void apply(List<ObservationHandler<?>> handlers, ObservationConfig config) {
	MultiValueMap<Class<? extends ObservationHandler>, ObservationHandler<?>> groupings = new LinkedMultiValueMap<>();
	for (ObservationHandler<?> handler : handlers) {
	Class<? extends ObservationHandler> category = findCategory(handler);
	if (category != null) {
	groupings.add(category, handler);
	}
	else {
	config.observationHandler(handler);
	}
	}
	for (Class<? extends ObservationHandler> category : this.categories) {
	List<ObservationHandler<?>> handlerGroup = groupings.get(category);
	if (!CollectionUtils.isEmpty(handlerGroup)) {
	config.observationHandler(new FirstMatchingCompositeObservationHandler(handlerGroup));
	}
	}
	}

The apply() method checks the category. If the handler is in a category, it is added to a handleGroup list. Those handlerGroups will be registered with FirstMatchingCompositeObservationHandler in the second for-loop. However, since my custom handler is not in a category, regardless I specify bean order or not, it is registered to the observation registry right away before the handlerGroups which contains DefaultTracingObservationHandler.

This ordering causes a problem.

The handler registration order becomes:

1. my custom handler
2. FirstMatchingCompositeObservationHandler which contains DefaultTracingObservationHandler.

When an observation starts, it goes through the handlers by the registered order, then on the stop, it goes by reverse order.

Since onStop is called in reverse order, DefaultTracingObservationHandler is called first and closes the current span. Then my custom handler tries to add key-values to the observation, but it will have no effect on the span since it was already closed.

A workaround is to use ObservationRegistryCustomizer bean to register my custom handler. It is because customizers are applied after observation handlers have registered.

I believe an observation handler that is not in any category needs to be registered after infrastructure handlers are registered.
Or, need a mechanism to choose whether to register a handler before/after handlerGroups registration.

[mhalbritter]

We now register uncategorized handlers after the categorized ones.