fix(gcpkms): Set quota project to API project

[onjen]

Like described in the linked issue, if the GCP KMS key is stored in project foo, but the service account is created in project bar, sops complains that KMS API is not enabled in project bar.

The quota project used by default is the one encoded in the service account key. With this commit, the behavior changes, so the project where the KMS key and API reside, is read from the key ID and set via the quota project option.

Fixes #1142
Fixes #2020

[rsalmond]

This is a lovely PR, bummed to find it sitting unmerged after hitting this same issue.

Anything I can do to help?

[onjen]

I just rebased, the change is waiting for someone to review and merge :)

[rsalmond]

Kk maybe this will help. Here's a manual test of this fix.

Before

$ sops --version
sops 3.11.0 (latest)

[warning] Note that in a future version, sops will no longer check whether the current version is the latest when asking for sops' version. If you want to explicitly check for the latest version, add the `--check-for-updates` option to `sops --version`. This will hide this deprecation warning and will always check, even if the default behavior changes in the future.

$ cat ~/.config/gcloud/application_default_credentials.json | jq '.quota_project_id'
"gpo-eng-stage"

^^ as noted by @haizaar on #1142 - if the application default creds have a quota project ID set, sops will execute the KMS API call against this GCP project.

$ cat .sops.yaml
creation_rules:
  - path_regex: secrets.stage.env
    gcp_kms: 'projects/gpo-bootstrap-2/locations/global/keyRings/sops-stage/cryptoKeys/sops-stage'

^^ my sops config references a key in a project called gpo-bootstrap-2, not my quota project.

$ sops encrypt secrets.stage.env
Could not generate data key: [failed to encrypt new data key with master key "projects/gpo-bootstrap-2/locations/global/keyRings/sops-stage/cryptoKeys/sops-stage": failed to encrypt sops data key with GCP KMS key: rpc error: code = PermissionDenied desc = Cloud Key Management Service (KMS) API has not been used in project gpo-eng-stage before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=gpo-eng-stage then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
error details: name = ErrorInfo reason = SERVICE_DISABLED domain = googleapis.com metadata = map[activationUrl:https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=gpo-eng-stage consumer:projects/gpo-eng-stage containerInfo:gpo-eng-stage service:cloudkms.googleapis.com serviceTitle:Cloud Key Management Service (KMS) API]
error details: name = Help desc = Google developers console API activation url = https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=gpo-eng-stage
error details: name = LocalizedMessage locale = en-US msg = Cloud Key Management Service (KMS) API has not been used in project gpo-eng-stage before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=gpo-eng-stage then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.]

^^ since the KMS API is not enabled in my default quota project, the call fails.

After

$ git fetch origin pull/1697/head:pr-1697

remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 4 (delta 2), reused 4 (delta 2), pack-reused 0 (from 0)
Unpacking objects: 100% (4/4), 723 bytes | 144.00 KiB/s, done.
From https://github.com/getsops/sops
 * [new ref]             refs/pull/1697/head -> pr-1697

$ git checkout pr-1697
Previous HEAD position was 168736311 Merge pull request #1960 from felixfontein/release-3.11.0
Switched to branch 'pr-1697'

$ go mod vendor
$ make install
$ ls -alh $(which sops)
-rwxr-xr-x 1 rsa staff 64M Dec  6 14:04 /Users/rsa/go/bin/sops

$ cat ~/.config/gcloud/application_default_credentials.json | jq '.quota_project_id'
"gpo-eng-stage"

^^ nothing up my sleeve, I haven't removed the quota project from my config.

$ sops encrypt secrets.stage.env
FOO=ENC[AES256_GCM,data:Vp/O,iv:dUg3Fk08wmuLQO+frKtGxtRjO4L4NRjC7LRZc18SBr0=,tag:51dTO7CKQaGXQSwzpaG5dg==,type:str]
sops_gcp_kms__list_0__map_created_at=2025-12-06T19:04:48Z
sops_gcp_kms__list_0__map_enc=CiQA2wUFSfdnLQ9Jp8h9aHw5bpIkGIuWGIBQrZ/dCyTfBqoNy1ASSQAv0ygTdz3jBcOLwg6DouLFM2per/DBFwbQ0I4LUX9/VUOuqt79GZ2YuUvpFED06ARnlhEcXvZTzQ8Y/G0M9n9aYxuVED4x6cg=
sops_gcp_kms__list_0__map_resource_id=projects/gpo-bootstrap-2/locations/global/keyRings/sops-stage/cryptoKeys/sops-stage
sops_lastmodified=2025-12-06T19:04:48Z
sops_mac=ENC[AES256_GCM,data:yyU4JSe0/EFpkGtG3bdUCdJ5mLIIqw0nF+MOjT0QewygNGVWJGsptoljN43ZKSTj4OdZla2ocCaNfk5CYRQESU8zJv6d+UWoG2zrvi2MzFA8TNwuhtObbftUWRqAdW+4xxcqp6EuXhS1UJQ+Qj1+1GLSU6hUoifN9Hw1l/rRdBg=,iv:IJA/4qJGyM6lPg2MerW5tEhVwt71wvMvS2Qfe3PCZMQ=,tag:b4I92AeKh1OJSt505BwObA==,type:str]
sops_unencrypted_suffix=_unencrypted
sops_version=3.11.0

🚀

[sabre1041]

LGTM

[felixfontein]

@onjen @rsalmond @sabre1041 thanks for fixing this!

[moritzschmitz-oviva]

We use sops in our company and with v3.12.1 we now see the following error:

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  projects/REDACTED/locations/europe-west3/keyRings/sops-hsm/cryptoKeys/hb-it: FAILED
    - | failed to decrypt sops data key with GCP KMS key: rpc error:
      | code = PermissionDenied desc = Caller does not have required
      | permission to use project REDACTED. Grant the caller the
      | roles/serviceusage.serviceUsageConsumer role, or a custom
      | role with the serviceusage.services.use permission, by
      | visiting
      | https://console.developers.google.com/iam-admin/iam/project?project=REDACTED
      | and then retry. Propagation of the new permission may take a
      | few minutes.
      | error details: name = ErrorInfo reason = USER_PROJECT_DENIED
      | domain = googleapis.com metadata =
      | map[consoleUrl:https://console.developers.google.com/iam-admin/iam/project?project=REDACTED
      | consumer:projects/REDACTED containerInfo:REDACTED
      | service:cloudkms.googleapis.com]
      | error details: name = Help desc = Google developer console
      | IAM admin url =
      | https://console.developers.google.com/iam-admin/iam/project?project=REDACTED
      | error details: name = LocalizedMessage locale = en-US msg =
      | Caller does not have required permission to use project
      | REDACTED. Grant the caller the
      | roles/serviceusage.serviceUsageConsumer role, or a custom
      | role with the serviceusage.services.use permission, by
      | visiting
      | https://console.developers.google.com/iam-admin/iam/project?project=REDACTED
      | and then retry. Propagation of the new permission may take a
      | few minutes.

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

Our .sops.yaml:

stores:
  yaml:
    indent: 2
creation_rules:
  - path_regex: hb/it/.*
    gcp_kms: projects/REDACTED/locations/europe-west3/keyRings/sops-hsm/cryptoKeys/hb-it

This is what I run locally for the default credentials:

gcloud auth application-default login --scopes=https://www.googleapis.com/auth/chat.messages,openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/chat.messages.create

And this returns null:

cat ~/.config/gcloud/application_default_credentials.json | jq '.quota_project_id'

This was working prior to v3.12.1 and I fail to understand how the behavior now requires the serviceUsageConsumer role.

@onjen or @sabre1041, do you have time to have a look here?

[rsalmond]

I couldn't tell you why it was working previously, but I know that the serviceusage.serviceUsageConsumer role is required when a service account in one GCP project needs to consume API's (in this case KMS) in another project.

[Dougniel]

I have the same issue than @moritzschmitz-oviva, fails since v3.12.1 :

• service account and the GCP KMS key are in the same project
• no quota_project_id defined int the ~/.config/gcloud/application_default_credentials.json

[rsalmond]

Heya, I'm not a maintainer (or even a contributor) to sops but I sorta kinda understand how it works with GCP. If you wanna open a new issue with steps to repro (help me get a brand new GCP project into a state that matches your environment) and tag me on it and I'll spend some time next week trying to investigate.

[moritzschmitz-oviva]

@rsalmond, I created a new issue here: #2088 (comment). Just FYI 😆. I added a guide on how to reproduce at the end.

[felixfontein]

We reverted the change for now; there's now a discussion #2108 about how the behavior should be.