terraform 1.1.0: fatal: Remote branch <GIT SHA> not found in upstream origin

[andresusanto]

Terraform Version

1.1.0

Terraform Configuration Files

module "some_module_name" {
  source           = "git@github.com:andresusanto/repo//modules/some_module?ref=923e51e12c6e5a1c76dd42a59837d5359d6eaa92" # a valid git commit sha

  some_variable = var.some_variable
}

Expected Behavior

Terraform should be able to pull the module from the repo. I have confirmed that it works on terraform 1.0.11.

Actual Behavior

╷
│ Error: Failed to download module
│ 
│ Could not download module "some_module_name"
│ (../modules/test/main.tf:8) source code from
│ "git::ssh://git@github.com/andresusanto/test-repo?ref=923e51e12c6e5a1c76dd42a59837d5359d6eaa92":
│ error downloading
│ 'ssh://git@github.com/andresusanto/test-repo?ref=923e51e12c6e5a1c76dd42a59837d5359d6eaa92':
│ /usr/bin/git exited with 128: Cloning into
│ '.terraform/modules/env.some_module_name'...
│ Warning: Permanently added the ECDSA host key for IP address '140.82.112.4'
│ to the list of known hosts.
│ fatal: Remote branch 923e51e12c6e5a1c76dd42a59837d5359d6eaa92 not found in
│ upstream origin

Steps to Reproduce

Should be able to reproduce this by using any module with git source and ref pointing to commit sha on terraform 1.1.0.

[apparentlymart]

Thanks for reporting this, @andresusanto!

Based entirely on the message (and not any detailed debugging yet) it sounds like Terraform tried to run git fetch ssh://git@github.com/andresusanto/test-repo 923e51e12c6e5a1c76dd42a59837d5359d6eaa92 (or a similar command using git clone with the branch option), rather then just cloning everything and then switching to the branch afterwards. I don't believe we intentionally changed anything in this way but we should investigate whether perhaps our upstream dependency go-getter has introduced what might've been intended as an optimization (clone only the requested branch) but that doesn't work for anything that isn't a named ref.

The other possibility here is that the earlier normalization of module source addresses might account for that, but at first glance I can't yet see why that would cause this effect, since clearly the normalization did preserve the ref=... argument exactly as you wrote it.

We'll look into this more closely and see if we can reproduce it and then explain what's changed. Thanks again!

[andresusanto]

Cheers @apparentlymart !

[scott-doyland-burrows]

Same for me, my root module looks like this - so it is just requesting the latest commit.
Reverted to 1.0.11 and it works fine.

module "aws_instance" {
  source = "git::https://github.com/[redacted]/terraform-aws-hgrs-ec2.git"

Error: Failed to download module

Could not download module "aws_instance" (ec2instance.tf:1) source code from
"git::https://github.com/[redacted]/terraform-aws-hgrs-ec2.git": error
downloading 'https://github.com/[redacted]/terraform-aws-hgrs-ec2.git':
/usr/bin/git exited with 128: Cloning into
'.terraform/modules/aws_instance'...
fatal: Remote branch master not found in upstream origin

[amit-disc]

Should we now always specify ref now? It is defaulting to master,

[apparentlymart]

Hi all,

This change in behavior is a bug, so I would not suggest changing your configuration to work around it unless you need to upgrade to Terraform v1.1.0 immediately for any reason. We intend to restore the previous behavior in a later release.

With that said, if you do need or want to upgrade now then setting ?ref=main (or whatever is the name of your default branch) may be a reasonable workaround, though I'm not sure yet because we've not yet completed a reproduction on our side.

Thanks!

[apparentlymart]

I was able to reproduce this against a repository we typically use for testing module installation, as follows:

module "example" {
  source = "git::https://github.com/hashicorp/terraform-aws-module-installer-acctest?ref=fbad92afe22792b939ceb233acd86ebd57af8fc7"
}

Initializing modules...
Downloading git::https://github.com/hashicorp/terraform-aws-module-installer-acctest?ref=fbad92afe22792b939ceb233acd86ebd57af8fc7 for example...
╷
│ Error: Failed to download module
│ 
│ Could not download module "example" (module-source-git.tf:2) source code from
│ "git::https://github.com/hashicorp/terraform-aws-module-installer-acctest?ref=fbad92afe22792b939ceb233acd86ebd57af8fc7": error downloading
│ 'https://github.com/hashicorp/terraform-aws-module-installer-acctest?ref=fbad92afe22792b939ceb233acd86ebd57af8fc7': /usr/bin/git exited with 128: Cloning into
│ '.terraform/modules/example'...
│ fatal: Remote branch fbad92afe22792b939ceb233acd86ebd57af8fc7 not found in upstream origin
│ 
╵

However, I was not able to reproduce with a URL that lacked any ref at all:

module "example" {
  source = "git::https://github.com/hashicorp/terraform-aws-module-installer-acctest"
}

$ terraform init
Initializing modules...
Downloading git::https://github.com/hashicorp/terraform-aws-module-installer-acctest for example...
- example in .terraform/modules/example
- example.child_a in .terraform/modules/example/modules/child_a
- example.child_a.child_b in .terraform/modules/example/modules/child_b

Initializing the backend...

Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Since this issue was originally about selecting an arbitrary commit instead of a named ref, I'm going to focus on that part (which I've now reproduced) for the moment, but if I learn something along the way that might explain why a "ref-less" source address would return a similar error then I will aim to fix that too. In the meantime, if anyone else can confirm that particular behavior with no ref argument at all, I am happy to retry reproduction of that situation with new information.

[scott-doyland-burrows]

It will fail without the ref:

terraform {
  required_providers {
    aws = {
    }
  }
}

provider "aws" {
  region = "eu-west-2"
}


module "s3bucket" {
  source = "git::https://github.com/[redacted]/terraform-aws-hgrs-s3bucket.git"

  s3bucket_bucket = "qwerty123456"
}

This works fine if I run a terraform init from the terraform CLI manually - I see this:

Terraform v1.1.0
on linux_amd64
doylands@AZWEWSH5DB933:~/temp1$ terraform init
Initializing modules...
Downloading git::https://github.com/[redacted]/terraform-aws-hgrs-s3bucket.git for s3bucket...
- s3bucket in .terraform/modules/s3bucket

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v3.68.0...
- Installed hashicorp/aws v3.68.0 (signed by HashiCorp)

It fails if I use the same code but call it via GitHub Actions - my GitHub Actions workflow code is:

name: test
on:
  workflow_dispatch:
env:
  TF_IN_AUTOMATION: true
  TF_INPUT: false
  TF_REGION: eu-west-2
  AWS_ACCOUNT_ACCESS_KEY: ${{ secrets.[redacted] }}
  AWS_ACCOUNT_SECRET_KEY: ${{ secrets.[redacted] }}
jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: hashicorp/setup-terraform@v1
        with:
          terraform_wrapper: false
      - uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ env.AWS_ACCOUNT_ACCESS_KEY }}
          aws-secret-access-key: ${{ env.AWS_ACCOUNT_SECRET_KEY }}
          aws-region: ${{ env.TF_REGION }}
      - run: |
          PAT=`aws secretsmanager get-secret-value --secret-id [redacted] --query SecretString --output text`
          git config --global url."https://$PAT@github.com".insteadOf https://github.com
          terraform version
          terraform init

and the error is:

Terraform v1.1.0
on linux_amd64
Upgrading modules...
Downloading git::https://github.com/[redacted]/terraform-aws-hgrs-s3bucket.git for s3bucket...

Error: Failed to download module

Could not download module "s3bucket" (terraform.tf:13) source code from
"git::https://github.com/[redacted]/terraform-aws-hgrs-s3bucket.git": error
downloading 'https://github.com/[redacted]/terraform-aws-hgrs-s3bucket.git':
/usr/bin/git exited with 128: Cloning into '.terraform/modules/s3bucket'...
fatal: Remote branch master not found in upstream origin

If I alter my workflow code to specify the older terraform release, then it works:

- uses: hashicorp/setup-terraform@v1
        with:
          terraform_wrapper: false
          terraform_version: 1.0.11

[apparentlymart]

I have investigated the changed in behavior and I believe I've narrowed it down to a change in behavior in our upstream library go-getter, which handles module installation from git.

Terraform v1.0.11 was using go-getter v1.5.2 while v1.1.0 is now using v1.5.9. New in that upgrade is hashicorp/go-getter#266, which added some extra logic to better handle the shallow clone optimization previously added, but it seems to have come with the assumption that ref will always be a named ref and never a standalone commit, which is a breaking change from Terraform's perspective.

In particular, it changes the git clone command with a commit to be like the following, which reproduces the error reported here:

$ git clone --branch fbad92afe22792b939ceb233acd86ebd57af8fc7 https://github.com/hashicorp/terraform-aws-module-installer-acctest
Cloning into 'terraform-aws-module-installer-acctest'...
fatal: Remote branch fbad92afe22792b939ceb233acd86ebd57af8fc7 not found in upstream origin

---

It also introduced a new step to infer the default branch in the case where no ref is specified, which I've not yet proven is related to the other behavior of failing when no ref is specified but I have a theory that we can now hopefully test.

If you don't specify ref then go-getter will ask the remote repository an additional question to learn which branch is the default branch:

$ git ls-remote --symref https://github.com/hashicorp/terraform-aws-module-installer-acctest HEAD
ref: refs/heads/master  HEAD
fbad92afe22792b939ceb233acd86ebd57af8fc7        HEAD

In the case of the repository I'm testing with here, the default branch is named master and that branch actually exists. However, since HEAD is just a symbolic ref (that is, a file in the remote repository containing the string ref: refs/heads/master), there's no guarantee that it will always refer to a branch that actually exists. In particular, if you'd previously renamed the main branch to have another name but didn't update HEAD then you could potentially have that symbolic ref still pointing at the now-obsolete name "master", leading to a subsequent failure trying to clone from that branch.

@scott-doyland-burrows, can you try a command similar to the git ls-remote command I showed above against your repository, verify that it did indeed indicate refs/heads/master, and confirm whether your repository has a branch named master? Thanks!