Sync JWKS URI

[h2zh]

This PR syncs the JWKS URI in federation discovery endpoint /.well-known/pelican-configuration with the JWKS URI in Pelican process' global federation metadata pelican_url.FederationDiscovery

• Before this PR, the JWKS URI in federation discovery endpoint is just appending /.well-known/pelican-configuration to Director URL, even though the Discovery URL is set. This means the JWKS URI provided by Discovery URL may contain a different set of keys.
  But this may be intentional! Director should advertise its own key endpoint rather than potentially pointing to a different server's JWKS URI.

A quick recap: In discoverFederationImpl func, fedInfo.JwksUri will first fill with the value of the Federation.JwkUrl config param. If it is not set, that function will probe Discovery Endpoint to get the jwks_uri there. At last, if fedInfo.JwksUri is still empty, it will fallbacks to Director's externalUrl + "/.well-known/issuer.jwks"

• This PR will work with Periodic metadata discrepancy check for the Director #3008. How does they work?
• This PR was intended to replace Periodic metadata discrepancy check for the Director #3008. But I realize maybe the good practice for a server is using its own JWKS URI, instead of pointing to another's, even though there might be discrepancy for the keys between them! We should still use a dedicated goroutine to monitor the discrepancy, as in Periodic metadata discrepancy check for the Director #3008. So I just close this PR.

[jhiemstrawisc]

I know you closed this, but one thing to consider -- the /.well-known/pelican-configuration path is always meant to be the Pelican Federation's metadata discovery endpoint, and I really do think it should be globally unique. When the Director needs to advertise its own keys, it should do so through the standard OIDC flow, which puts the jwks_uri behind /.well-known/openid-configuration. This lets us keep the two separate so that we can guarantee all copies of /.well-known/pelican-configuration are the same.

[h2zh]

Thanks for the enlightenment @jhiemstrawisc. I think it would be better to reopen this PR and close the other one (#3008)?

[jhiemstrawisc]

Up to you how you want to resolve things! I'd argue what we're shooting for here is:

1. If a Director hosts a copy of metadata at /.well-known/pelican-configuration, it should match exactly any metadata hosted at the Federation's discovery URL. I think your other PR takes the right approach in displaying discrepancies without making this the kind of error that prevents you from starting the Director.
2. Ideally, the Director is going through the same discovery process as anyone else and querying <discoveryUrl>/.well-known/pelican-configuration to make a copy of it directly. The only tricky part might be if the Director has a config like:

Federation:
  DiscoveryUrl: osg-htc.org
  RegistryUrl: some-hostname-different-from-what-gets-discovered-at-osg-htc.org

Since we technically allow admins to overwrite discovered values with whatever they want via config, we should ask whether a configuration like that means a) "Use this different Registry internally, but keep the duplicate fed discovery metadata consistent by using the discovered registry" or does it mean b) "I want to use this Registry and also overwrite the value that gets put into federation metadata discovery. While I'm leaning towards a), I think for now it's better to do b) and present the warning you worked on in #3008 (otherwise you'll have to decouple big sections of code). What do you think?

[h2zh]

After one day of consideration, my plan is moving forward with both this PR and #3008.

This PR can ensure the same keys are provided via the /.well-known/pelican-configuration endpoints at Director and discovery URL (just like the 1. you mentioned).

And #3008 can warn the Director admin for rare scenarios when they overwrite the config related to the metadata. Beside your example in 2., they can customize the jwks_uri to serve a different set of keys.

Federation:
  JwkUrl: <>

[jhiemstrawisc]

I almost wonder if this should be an error that prevents the Director from starting -- if there's no jwks_uri, can the federation function at all? I'll let you decide how far you want to chase it.

[h2zh]

I chased to the source and found out that jwksUri could never be empty for Director, which means the "empty jwksUri" error handling section in this PR will never be triggered.

For Director servers, jwksUri will never be empty in practice because Server_ExternalWebUrl always has a value—it defaults to https://<hostname>:<port> (computed from os.Hostname() and Server.WebPort) if not explicitly configured (see ComputeExternalWebUrl() in config.go).

Since externalUrlStr is populated from Server_ExternalWebUrl.GetString() at the start of this function, the fallback assignment in the defer block will always execute successfully:

func discoverFederationImpl(ctx context.Context) (fedInfo pelican_url.FederationDiscovery, err error) {
	federationStr := param.Federation_DiscoveryUrl.GetString()
	externalUrlStr := param.Server_ExternalWebUrl.GetString()
	defer func() {
		...
		if fedInfo.JwksUri == "" && enabledServers.IsEnabled(server_structs.DirectorType) {
			fedInfo.JwksUri = externalUrlStr + "/.well-known/issuer.jwks"
		}

[jhiemstrawisc]

Failing CI are known flaky tests, so I'm okay merging with these red boxes.