panic: didn't find any token of type TokenIdent

[jbmcfarlin31]

On another repo I tried this tool on, it worked wonderfully! I decided to try it on another branch of one of my other repos and it failed with the following:

panic: didn't find any token of type TokenIdent

goroutine 1 [running]:
github.com/hashicorp/hcl/v2/hclwrite.inputTokens.PartitionType(0xc00010cba0, 0x0, 0x0, 0xc0001160f8, 0x0, 0x0, 0x49, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:88 +0x580
github.com/hashicorp/hcl/v2/hclwrite.inputTokens.PartitionTypeSingle(0xc00010cba0, 0x0, 0x0, 0xc0001160f8, 0x0, 0x0, 0xc000000049, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:92 +0xe3
github.com/hashicorp/hcl/v2/hclwrite.parseTraversalStep(0x709140, 0xc00009aaa0, 0xc00010cba0, 0x0, 0x0, 0xc0001160f8, 0x0, 0x0, 0xc00010cba0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:395 +0x364
github.com/hashicorp/hcl/v2/hclwrite.parseTraversal(0xc00000f0c0, 0x2, 0x2, 0xc00010cba0, 0x0, 0x0, 0xc0001160f8, 0x0, 0x0, 0xc00010cba0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:377 +0x3da
github.com/hashicorp/hcl/v2/hclwrite.parseExpression(0x70a780, 0xc0000a2160, 0xc00010cba0, 0x0, 0x0, 0xc0001160f8, 0x0, 0x0, 0x8)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:356 +0x175
github.com/hashicorp/hcl/v2/hclwrite.parseAttribute(0xc0000b47e0, 0xc00010cba0, 0x5, 0x5, 0xc0001160f8, 0x5, 0x5, 0xc00010ca80, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:260 +0xaf4
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x707960, 0xc0000b47e0, 0xc00010ca80, 0x46, 0x46, 0xc0001160e0, 0x46, 0x46, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:217 +0x632
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc0000a2370, 0xc00010ca80, 0x46, 0x46, 0xc0001160e0, 0x46, 0x46, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:193 +0x68d
github.com/hashicorp/hcl/v2/hclwrite.parseBlock(0xc00009e240, 0xc00010ca80, 0x0, 0x0, 0xc0001160e0, 0x0, 0x0, 0xc00010c840, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:329 +0xf72
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x7079a0, 0xc00009e240, 0xc00010c840, 0xd0, 0xd0, 0xc0001160b0, 0xd0, 0xd0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:219 +0x4f7
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc0000a2790, 0xc00010c360, 0xdd, 0xdd, 0xc000116048, 0xdd, 0xdd, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:193 +0x68d
github.com/hashicorp/hcl/v2/hclwrite.parseBlock(0xc00009ea20, 0xc00010c360, 0x0, 0x0, 0xc000116048, 0x0, 0x0, 0xc00010c000, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:329 +0xf72
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x7079a0, 0xc00009ea20, 0xc00010c000, 0xe8, 0xe8, 0xc000116000, 0xe8, 0xe8, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:219 +0x4f7
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc0000a2840, 0xc00010c000, 0xe9, 0x100, 0xc000116000, 0xe9, 0xe9, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:193 +0x68d
github.com/hashicorp/hcl/v2/hclwrite.parse(0xc0000f2000, 0x769, 0x969, 0xc0000201c0, 0x34, 0x1, 0x1, 0x0, 0xffffffffffffff9c, 0xc000020200, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:53 +0x2e4
github.com/hashicorp/hcl/v2/hclwrite.ParseConfig(...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/public.go:27
main.processFile(0xc0000201c0, 0x34, 0x1a4)
        /tmp/terraform-clean-syntax/main.go:78 +0xc8
main.processItem(0xc0000201c0, 0x34)
        /tmp/terraform-clean-syntax/main.go:55 +0x2e7
main.processDir(0x7ffcdd7ba68a, 0x2d)
        /tmp/terraform-clean-syntax/main.go:67 +0x119
main.processItem(0x7ffcdd7ba68a, 0x2e)
        /tmp/terraform-clean-syntax/main.go:47 +0x1cc
main.main()
        /tmp/terraform-clean-syntax/main.go:30 +0xec

I re-cloned this repo, rebuilt the binary, and tried again but nothing changed, still the same error.
terraform-clean /my/dir/

Thanks!

[apparentlymart]

Hi @jbmcfarlin31!

885338e should've fixed issues of this type, but perhaps you've found a new problem with similar symptoms.

If you rebuild from the latest code on master then you'll get the change from #6 that should allow the program to complete in spite of this crash, and so at least it should be able to update the other files that don't run into this problem. If you're then able to share the source code of the file that is indicated as the cause of the crash, hopefully we can figure out what part of the input is not being handled correctly by the parser.

[jbmcfarlin31]

Hi @apparentlymart looks like this is happening with a third party provider. I did a recent pull of this repo and rebuilt the binary. Then I tested in my branch one file at a time until I found the culprit.

The error I receive is the following:

Recovered in processFile while processing rke.tf: "didn't find any token of type TokenIdent"
goroutine 1 [running]:
runtime/debug.Stack(0xc00011b970, 0x624e40, 0xc000124160)
        /root/go/src/runtime/debug/stack.go:24 +0x9d
main.processFile.func1(0xc000014508, 0x6)
        /tmp/terraform-clean-syntax/main.go:82 +0x6e
panic(0x624e40, 0xc000124160)
        /root/go/src/runtime/panic.go:522 +0x1b5
github.com/hashicorp/hcl/v2/hclwrite.inputTokens.PartitionType(0xc0000f6ba0, 0x0, 0x0, 0xc0001220f8, 0x0, 0x0, 0x49, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:88 +0x57d
github.com/hashicorp/hcl/v2/hclwrite.inputTokens.PartitionTypeSingle(0xc0000f6ba0, 0x0, 0x0, 0xc0001220f8, 0x0, 0x0, 0xc000000049, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:92 +0xe3
github.com/hashicorp/hcl/v2/hclwrite.parseTraversalStep(0x6fa820, 0xc00007c910, 0xc0000f6ba0, 0x0, 0x0, 0xc0001220f8, 0x0, 0x0, 0xc0000f6ba0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:395 +0x365
github.com/hashicorp/hcl/v2/hclwrite.parseTraversal(0xc00000ee60, 0x2, 0x2, 0xc0000f6ba0, 0x0, 0x0, 0xc0001220f8, 0x0, 0x0, 0xc0000f6ba0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:377 +0x3dc
github.com/hashicorp/hcl/v2/hclwrite.parseExpression(0x6fbca0, 0xc0000822c0, 0xc0000f6ba0, 0x0, 0x0, 0xc0001220f8, 0x0, 0x0, 0x8)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:356 +0x178
github.com/hashicorp/hcl/v2/hclwrite.parseAttribute(0xc0000982a0, 0xc0000f6ba0, 0x5, 0x5, 0xc0001220f8, 0x5, 0x5, 0xc0000f6a80, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:260 +0xafe
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x6f9040, 0xc0000982a0, 0xc0000f6a80, 0x46, 0x46, 0xc0001220e0, 0x46, 0x46, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:217 +0x632
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc000082420, 0xc0000f6a80, 0x46, 0x46, 0xc0001220e0, 0x46, 0x46, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:193 +0x6bb
github.com/hashicorp/hcl/v2/hclwrite.parseBlock(0xc000080240, 0xc0000f6a80, 0x0, 0x0, 0xc0001220e0, 0x0, 0x0, 0xc0000f6840, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:329 +0xf7f
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x6f9080, 0xc000080240, 0xc0000f6840, 0xd0, 0xd0, 0xc0001220b0, 0xd0, 0xd0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:219 +0x4f7
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc000082840, 0xc0000f6360, 0xdd, 0xdd, 0xc000122048, 0xdd, 0xdd, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:193 +0x6bb
github.com/hashicorp/hcl/v2/hclwrite.parseBlock(0xc000080900, 0xc0000f6360, 0x0, 0x0, 0xc000122048, 0x0, 0x0, 0xc0000f6000, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:329 +0xf7f
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x6f9080, 0xc000080900, 0xc0000f6000, 0xe8, 0xe8, 0xc000122000, 0xe8, 0xe8, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:219 +0x4f7
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc0000828f0, 0xc0000f6000, 0xe9, 0x100, 0xc000122000, 0xe9, 0xe9, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:193 +0x6bb
github.com/hashicorp/hcl/v2/hclwrite.parse(0xc0000c4000, 0x765, 0x965, 0xc000014508, 0x6, 0x1, 0x1, 0x0, 0x490beb, 0x106, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:53 +0x2e4
github.com/hashicorp/hcl/v2/hclwrite.ParseConfig(...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/public.go:27
main.processFile(0xc000014508, 0x6, 0x1a4)
        /tmp/terraform-clean-syntax/main.go:86 +0x119
main.processItem(0xc000014508, 0x6)
        /tmp/terraform-clean-syntax/main.go:57 +0x2ea
main.processDir(0x7ffc2fa5368b, 0x1)
        /tmp/terraform-clean-syntax/main.go:69 +0x11a
main.processItem(0x7ffc2fa5368b, 0x1)
        /tmp/terraform-clean-syntax/main.go:49 +0x1cc
main.main()
        /tmp/terraform-clean-syntax/main.go:32 +0xea

The terraform code in question is:

# ---------------------------------------------------------------------
# RKE configuration

resource rke_cluster "cluster" {
  depends_on = [azurestack_public_ip.vmpip, azurestack_virtual_machine.vm]

  dynamic "nodes" {
    for_each = azurestack_public_ip.vmpip.*
    iterator = nodes

    content {
      address = nodes.value.ip_address
      user    = "testuser"
      #role    = nodes.key < var.nbr_managers ? ["controlplane","etcd"] : ["worker"]
      role    = ["controlplane","etcd", "worker"]
      ssh_key = file("/opt/${var.deployment_name}/${var.deployment_name}")
    }
  }

  ignore_docker_version = true
  cluster_name = "${var.deployment_name}-cluster"

  # Kubernetes version
  kubernetes_version = "v1.15.3-rancher1-1"

  private_registries {
    url      = "my.repo.com:1234"
  }

  #########################################################
  # Network(CNI) - supported: flannel/calico/canal/weave
  #########################################################
  # There are several network plug-ins that work, but we default to canal
  network {
    plugin = "weave"
  }

  ingress {
    provider = "nginx"
    extra_args = {
      default-ssl-certificate = "ingress-nginx/wildcard-ingress"
    }
  }

  services_kube_api {
    extra_args = {
      enable-admission-plugins = "ServiceAccount,PodPreset,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds"
      runtime-config = "batch/v2alpha1,authentication.k8s.io/v1beta1=true,settings.k8s.io/v1alpha1=true"
    }
  }

  services_kubelet {
    extra_args = {
      volume-plugin-dir = "/usr/libexec/kubernetes/kubelet-plugins/volume/exec"
    }
    # Optionally define additional volume binds to a service
    extra_binds = [
      "/usr/libexec/kubernetes/kubelet-plugins/volume/exec:/usr/libexec/kubernetes/kubelet-plugins/volume/exec",
    ]
  }
}

It does look like the latest build of the binary allowed the other files to build despite having a panic. So that's a plus.