user.name is always UUID when using OIDC and Spring Security

[fgabolde]

Is your feature request related to a problem?

When using Spring Security oauth2 authentication to secure an API server, user.name in APM always shows the user ID (as it's the "name" of the Principal).

If a user describes an issue and I go into the APM service view to investigate, I don't typically have their UUID (and neither do they). I do however have their email (login) or preferred username, so I need to recover their UUID manually from Keycloak.

Describe the solution you'd like

I'd like a different claim, in case of JWTs, to be logged instead of or in addition to the UUID. preferred_username is already mentioned in an Azure SSO fallback. Ideally, which claim is used would be configurable. Of course this is only valid for JWTs and similar tokens, which is my use case, so not entirely generic.

This may require going through the Spring MVC plugin instead of the Servlet one?

Describe alternatives you've considered

Using MDC to get additional claims into the transaction data, but it doesn't fit well in the ECS schema and it requires adding code to every backend service.

[SylvainJuge]

Hi !

If I remember correctly, with APM instrumentation, the user.name is captured from the servlet principal. In your case, you get the user UUID from the implementation of your servlet authentication, which is clearly not ideal for investigation.

Capturing other claims from the JWT or any similar mechanism is technically feasible, but it's hard to get a generic enough interface to automatically instrument it, unlike the servlet principal which is a rather simple API.

I am not familiar enough with the Spring and Spring MVC framework internals to know how to change that, but the ideal steps to implement this would be to:

• find the method/classes to override to implement and call explicitly the agent public API (1)
• create/modify instrumentation to provide this automatically (2)

If you can provide an example for the first part (1), ideally with a simple spring application example it would really be great.

In addition, I think that using OpenTelemetry (or our EDOT agent distribution) could provide a few extra interesting features:

• using the baggage API we can add an attribute to the current transaction/span and also propagate to the downstream services, in your case it means you'll get the user name even on the database call spans.
• we could contribute this to upstream opentelemetry

Because you mention ECS switching to OpenTelemetry might be something challenging in your case, however I clearly see how the use-case your describe here would be relevant.

[fgabolde]

Hi !

If I remember correctly, with APM instrumentation, the user.name is captured from the servlet principal. In your case, you get the user UUID from the implementation of your servlet authentication, which is clearly not ideal for investigation.

Capturing other claims from the JWT or any similar mechanism is technically feasible, but it's hard to get a generic enough interface to automatically instrument it, unlike the servlet principal which is a rather simple API.

This matches exactly what I found after a quick look at the instrumentation code. The servlet instrumentation is super simple and clean... but the interop with Spring Security ends up with this user UUID in the principal, which is not flexible enough to represent complex auth payloads like JWTs. Oh well, tradeoffs.

I am not familiar enough with the Spring and Spring MVC framework internals to know how to change that, but the ideal steps to implement this would be to:

* find the method/classes to override to implement and call explicitly the agent public API (1)

* create/modify instrumentation to provide this automatically (2)

If you can provide an example for the first part (1), ideally with a simple spring application example it would really be great.

You mean inject the username ourselves, in app code, not agent code? Something like this?

Transaction transaction = ElasticApm.currentTransaction();
transaction.setUser(theUuid, theEmail, theUsername);

I'll take a look, I'm sure I can manage a security filter that does this. Now my issue will be adding it to a dozen microservices :)

By the way, what is the domain parameter in the four arg version of the setUser method? Doc just says domain: The user’s domain or null, if not applicable..

In addition, I think that using OpenTelemetry (or our EDOT agent distribution) could provide a few extra interesting features:

* using the baggage API we can add an attribute to the current transaction/span and also propagate to the downstream services, in your case it means you'll get the user name even on the database call spans.

* we could contribute this to upstream opentelemetry

Because you mention ECS switching to OpenTelemetry might be something challenging in your case, however I clearly see how the use-case your describe here would be relevant.

I'm not familiar with EDOT after a couple years of using this agent and the classic APM stack, I need to brush up on the changes in your observability stack. Is this an agent (like this one) but that emits otel traces that otel collectors can read? We are considering switching to opentelemetry... sometime in the next couple of years... and my fear was having to redo instrumentation by hand after getting comfortable with the automagic instrumentation provided here, so this could help but I wouldn't aim for a switch this month so staying on apm-agent-java for the moment.

[SylvainJuge]

Yes, providing a security filter (or whatever spring security allows) would be the first step.

Then, we could improve that by trying to automatically do that by making the agent provide the implementation (so changes are not needed anymore in the app).

I don't recall what the domain means in this context nor how it's being used in practice, how would you try to use this ?

EDOT is just a distribution of the OpenTelemetry instrumentation, so you'll get similar automagic instrumentation. One difference though is that if you have some manual instrumentation already in place it would have to be migrated to the OpenTelemetry API instead. You can look at our migration documentation for more details: https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/java/migration

[fgabolde]

OK, a filter works fine. Here it is without context, as I still need to build a minimal Spring app:

package fakepackagename;

import co.elastic.apm.api.ElasticApm;
import co.elastic.apm.api.Transaction;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.oauth2.jwt.Jwt;
import org.springframework.security.oauth2.server.resource.authentication.JwtAuthenticationToken;
import org.springframework.web.filter.GenericFilterBean;

import java.io.IOException;
import java.util.Map;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.ServletRequest;
import jakarta.servlet.ServletResponse;

public class ApmExtraDataBean extends GenericFilterBean {
  @Override
  public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
      throws IOException, ServletException {
    Transaction transaction = ElasticApm.currentTransaction();
    Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
    if (authentication instanceof JwtAuthenticationToken jwtAuthenticationToken) {
      Jwt jwt = (Jwt) jwtAuthenticationToken.getPrincipal();
      Map<String, Object> claims = jwt.getClaims();
      // the sub claim is registered for the JWT spec
      // (https://auth0.com/docs/secure/tokens/json-web-tokens/json-web-token-claims), email and preferred_username are
      // OIDC extensions (https://openid.net/specs/openid-connect-core-1_0.html#StandardClaims)
      transaction.setUser(jwt.getSubject(), jwt.getClaimAsString("email"), jwt.getClaimAsString("preferred_username"));
    }
    chain.doFilter(request, response);
  }
}

This is OIDC specific as it makes use of OIDC claims (email, preferred_username). Oauth2 itself does not specify any claims beyond what JWT itself recommends, I believe it is more flexible in that regard so I'm not sure it's possible to predict where the "useful" claims would be. OIDC has more conventions so makes more sense to target here anyway.

Looking at the existing Spring MVC instrumentation, I'm not sure what needs to be done to inject this logic directly instead of relying on a filter bean. Authentication objects can be part of a controller method signature if a developer adds them, but they're not guaranteed. However SecurityContextHolder.getContext() might work even from advice code, I'm really not familiar enough with ByteBuddy to tell.

[SylvainJuge]

I see, this seems very specific to the implementations that you are using here. Defining a generic implementation for this would really be tricky and very likely brittle as those implementations are likely to change over time, for example the instanceof assumptions are good indicators for this.

As a side note, instead of using a servlet filter, do you know if there is a way to execute this directly when the authentication is being checked, for example when spring security sets the value returned by SecurityContextHolder.getContext().getAuthentication() ?

[fgabolde]

I see, this seems very specific to the implementations that you are using here. Defining a generic implementation for this would really be tricky and very likely brittle as those implementations are likely to change over time, for example the instanceof assumptions are good indicators for this.

Yes, it's also a pain from application code. While Spring Security takes care of all the oauth2 generic parts, actually reading the claims and doing authorization based on their values is left to this type of code. Basically we feel safe in reading claims like this because we know that we configured the app to accept only tokens from this known identity provider. We can change the auth method and provider using configuration, but we'd have to change this authz code to follow because if we switched to Basic auth for instance, the Authentication objects would have a different implementation and we'd have to use different code to read user attributes (or rely on different Spring Security features to read them for us).

Here, I've at least stuck to assumptions that an OIDC-compliant API server would make, and I haven't used custom claims.

As a side note, instead of using a servlet filter, do you know if there is a way to execute this directly when the authentication is being checked, for example when spring security sets the value returned by SecurityContextHolder.getContext().getAuthentication() ?

This is Spring Security, so I'm sure there is a dozen different ways of customizing authentication yes :) You might provide your own bearer token filter to replace this: https://github.com/spring-projects/spring-security/blob/main/oauth2/oauth2-resource-server/src/main/java/org/springframework/security/oauth2/server/resource/web/authentication/BearerTokenAuthenticationFilter.java#L152 or provide a custom security context holder strategy or...