Signing key cache ignores SecretAccessKey change, causing SignatureDoesNotMatch with S3-compatible providers

[poymanovm]

Description

The derived signing key cache in aws/signer/internal/v4/cache.go validates cache entries by (AccessKeyID, Date) but does not check SecretAccessKey. When a credential provider returns the same AccessKeyID with a rotated SecretAccessKey, the signer uses a stale derived key computed from the old secret, producing SignatureDoesNotMatch (HTTP 403) on every request after credential refresh.

Root Cause

In aws/signer/internal/v4/cache.go:

type derivedKey struct {
	AccessKey  string    // only AccessKeyID is stored
	Date       time.Time
	Credential []byte
	// SecretAccessKey is NOT stored or compared
}

func (s *derivedKeyCache) get(key string, credentials aws.Credentials, signingTime time.Time) ([]byte, bool) {
	cacheEntry, ok := s.retrieveFromCache(key)
	if ok && cacheEntry.AccessKey == credentials.AccessKeyID && isSameDay(signingTime, cacheEntry.Date) {
		return cacheEntry.Credential, true // stale key returned!
	}
	return nil, false
}

The cache hit condition is AccessKeyID matches && same calendar day. If a credential provider returns new temporary credentials with the same AccessKeyID but a different SecretAccessKey, the cache returns the signing key derived from the old secret.

Why this doesn't manifest with AWS STS

AWS STS always returns a unique AccessKeyID (format ASIA...) for each set of temporary credentials. So the AccessKeyID comparison naturally fails after credential refresh, forcing a new key derivation. This makes the bug invisible on real AWS.

Reproduction scenario

• Provider: S3-compatible storage (Yandex Cloud Object Storage) with STS credentials
• Client: rclone v1.73.1 (uses aws-sdk-go-v2 v1.39.6)
• Credential source: credential_process returning STS tokens with ~4h TTL
• Workload: Large multipart upload lasting 20+ hours

What happens:

1. Initial credentials work fine for ~4 hours
2. SDK detects expiration and calls credential_process again
3. New credentials have the same AccessKeyID but new SecretAccessKey and SessionToken
4. Signer cache hits on (AccessKeyID, Date) → returns stale derived key
5. All subsequent requests fail with SignatureDoesNotMatch (HTTP 403)
6. The new SessionToken is correctly included in requests (visible in x-amz-security-token header), but the signature is computed with the old secret
7. Retries also fail — the cache is poisoned for the rest of the calendar day

Evidence from logs:

• Authorization header shows the same AccessKeyID before and after refresh
• x-amz-security-token header changes (new token from credential_process)
• Canonical string format is identical — same SignedHeaders, same scope
• In-flight requests signed with old credentials continue to succeed
• All new requests after refresh fail, including different operation types (UploadPart, CreateMultipartUpload, AbortMultipartUpload)

Comparison with SDK v1

AWS SDK Go v1 (aws-sdk-go) does not have this issue. The v1 signer calls deriveSigningKey() on every request without caching:

// aws-sdk-go v1: aws/signer/v4/v4.go
func (ctx *signingCtx) buildSignature() {
	creds := deriveSigningKey(ctx.Region, ctx.ServiceName,
		ctx.credValues.SecretAccessKey, ctx.Time)
	// ...
}

rclone v1.65.2 (SDK v1) works correctly with the same credential provider and endpoint.

Suggested Fix

Store and compare SecretAccessKey in the cache entry:

type derivedKey struct {
	AccessKey  string
	SecretKey  string    // add this
	Date       time.Time
	Credential []byte
}

Update the get method:

if ok && cacheEntry.AccessKey == credentials.AccessKeyID &&
   cacheEntry.SecretKey == credentials.SecretAccessKey &&  // add this check
   isSameDay(signingTime, cacheEntry.Date) {

And update the add/put method to store credentials.SecretAccessKey in the new field.

This is a minimal, backwards-compatible change that preserves the cache optimization for the common case (same credentials within a day) while correctly invalidating on secret rotation.

Environment

• aws-sdk-go-v2: v1.39.6 (via rclone v1.73.1)
• OS: Windows Server (amd64)
• S3-compatible endpoint: Yandex Cloud Object Storage
• Credential source: credential_process → HashiCorp Vault → Yandex Cloud STS