GCP: Unable to decrypt files with SOPS 3.12.1. Asks for `serviceusage.services.use` permissions now

[moritzschmitz-oviva]

We use sops in our company and with v3.12.1 we now see the following error:

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  projects/REDACTED/locations/europe-west3/keyRings/sops-hsm/cryptoKeys/hb-it: FAILED
    - | failed to decrypt sops data key with GCP KMS key: rpc error:
      | code = PermissionDenied desc = Caller does not have required
      | permission to use project REDACTED. Grant the caller the
      | roles/serviceusage.serviceUsageConsumer role, or a custom
      | role with the serviceusage.services.use permission, by
      | visiting
      | https://console.developers.google.com/iam-admin/iam/project?project=REDACTED
      | and then retry. Propagation of the new permission may take a
      | few minutes.
      | error details: name = ErrorInfo reason = USER_PROJECT_DENIED
      | domain = googleapis.com metadata =
      | map[consoleUrl:https://console.developers.google.com/iam-admin/iam/project?project=REDACTED
      | consumer:projects/REDACTED containerInfo:REDACTED
      | service:cloudkms.googleapis.com]
      | error details: name = Help desc = Google developer console
      | IAM admin url =
      | https://console.developers.google.com/iam-admin/iam/project?project=REDACTED
      | error details: name = LocalizedMessage locale = en-US msg =
      | Caller does not have required permission to use project
      | REDACTED. Grant the caller the
      | roles/serviceusage.serviceUsageConsumer role, or a custom
      | role with the serviceusage.services.use permission, by
      | visiting
      | https://console.developers.google.com/iam-admin/iam/project?project=REDACTED
      | and then retry. Propagation of the new permission may take a
      | few minutes.

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

Our .sops.yaml:

stores:
  yaml:
    indent: 2
creation_rules:
  - path_regex: hb/it/.*
    gcp_kms: projects/REDACTED/locations/europe-west3/keyRings/sops-hsm/cryptoKeys/hb-it

This is what I run locally for the default credentials:

gcloud auth application-default login --scopes=https://www.googleapis.com/auth/chat.messages,openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/chat.messages.create

And this returns null:

cat ~/.config/gcloud/application_default_credentials.json | jq '.quota_project_id'

This was working prior to v3.12.1 and I fail to understand how the behavior now requires the serviceUsageConsumer role.

How to reproduce:
https://gist.github.com/moritzschmitz-oviva/760ba1eac850e44b004c5502756d5fb4

[felixfontein]

It seems that opion.WithQuotaProject() requirs the serviceusage.services.use permission (which is part of the roles/serviceusage.serviceUsageConsumer role, but also can be applied without all the other permissions that are part of roles/serviceusage.serviceUsageConsumer by creating a more specialized role - at least that's what the error message claims). This does make sense according to https://docs.cloud.google.com/service-usage/docs/access-control#permissions (at the bottom of the table):

To use a project for quota and billing purposes. For more information, see System parameters.	On the project: serviceusage.services.use

I guess that #1697 applied opion.WithQuotaProject() also in situations where it shouldn't, namely when the key is part of the same project. Since I'm not familiar with the GCP API at all, I don't know if there's a better solution, but it might be best to simply revert that PR and try to solve the problem the PR tries to fix in another way? Forcing all GCP users to add a new permission that only some need sounds like a bad idea.

CC @onjen @rsalmond @sabre1041

(This kind of reminds me of the aws_profile problem that SOPS' AWS KMS implementation has - see for example #1696.)

[Heleydin]

Hello,
I confirm we have the same issue within our company.
Both KMS key and Google service account are in the same project.

[onjen]

This is an unintended behavior introduced with #1697
Turns out setting the quota project sets the X-Goog-User-Project header in the API which always triggers the authorization check if the use has the serviceusage.services.use permissions. There is no distinction if this is the same project the caller would have been billed to anyway.

Forcing all GCP users to add a new permission that only some need sounds like a bad idea.

Yes I agree this is not a good solution.

If you want to have a quick fix, revert would be the way to go.
A potential fix for both uses cases (same project and cross project) would be to conditionally only set the quota project if the projects differ. For this the KMS project is extracted from the KMS ID and the credentials project from the credential JSON.

What do you think? I could create a PR for the project extraction and conditonal setting of the quota project.

[dharmab]

Hi, could this change be reverted?

Downgrading sops installed via Homebrew is not simple, and involves downloading 1.3GB of data (brew tap homebrew/core --force). This has a potentially large blast radius, I don't think users will expect this in a patch version.

[felixfontein]

@dharmab this was introduced in a minor release, not a patch release.

[AlxCloudRl]

We are facing the issue too at our company. We use SOPS from a consumer project (project A) that needs to use keys in a centralized KMS project (project B). We now need to grant the Service Usage Consumer role to users on project B. This was not required before, and we would like for the quotas to stay at the consumer project level, to avoid having many consumers projects consuming the same quota on the centralized project B.

[felixfontein]

I created a PR (#2099) to revert #1697 in case there's no better fix for this issue.