feat: Replace the local-exec script with a http datasource for waiting cluster

[barryib]

PR o'clock

Description

In this PR we drop the usage of local-exec. But we need hashicorp/terraform-provider-http#29.

We decided to fork the terraform-provider-http provider (the PR is almost a year) into the terraform-aws-modules organization for internal usage only. The provider is published at https://registry.terraform.io/providers/terraform-aws-modules/http/latest

Related also to #1253

Test

This is still a proof of concept. I tested it locally and so far so good. I've got this kind of output during my tests.

module.eks.aws_eks_cluster.this[0]: Creation complete after 10m20s [id=test-eks-kvnu9oYg]
module.eks.data.http.wait_for_cluster[0]: Reading...
module.eks.data.template_file.userdata[0]: Reading...
module.eks.data.template_file.userdata[1]: Reading...
module.eks.aws_iam_role.workers[0]: Creating...
module.eks.data.template_file.userdata[1]: Read complete after 0s [id=3ffad6f89915b07a9027e89ab9f36248b75b36c289057f594e79061ec808d3fb]
module.eks.data.template_file.userdata[0]: Read complete after 0s [id=3ffad6f89915b07a9027e89ab9f36248b75b36c289057f594e79061ec808d3fb]
module.eks.local_file.kubeconfig[0]: Creating...
module.eks.local_file.kubeconfig[0]: Creation complete after 0s [id=55314f623fda1f56705845677c8bbddf79c83f4e]
module.eks.aws_iam_role.workers[0]: Creation complete after 2s [id=test-eks-kvnu9oYg2021050708275921740000000c]
module.eks.aws_iam_role_policy_attachment.workers_AmazonEC2ContainerRegistryReadOnly[0]: Creating...
module.eks.aws_iam_role_policy_attachment.workers_AmazonEKS_CNI_Policy[0]: Creating...
module.eks.aws_iam_role_policy_attachment.workers_AmazonEKSWorkerNodePolicy[0]: Creating...
module.eks.aws_iam_instance_profile.workers[0]: Creating...
module.eks.aws_iam_instance_profile.workers[1]: Creating...
module.eks.aws_iam_role_policy_attachment.workers_AmazonEC2ContainerRegistryReadOnly[0]: Creation complete after 1s [id=test-eks-kvnu9oYg2021050708275921740000000c-2021050708280162970000000f]
module.eks.aws_iam_role_policy_attachment.workers_AmazonEKS_CNI_Policy[0]: Creation complete after 1s [id=test-eks-kvnu9oYg2021050708275921740000000c-20210507082801643400000010]
module.eks.aws_iam_role_policy_attachment.workers_AmazonEKSWorkerNodePolicy[0]: Creation complete after 1s [id=test-eks-kvnu9oYg2021050708275921740000000c-20210507082801988200000011]
module.eks.aws_iam_instance_profile.workers[1]: Creation complete after 2s [id=test-eks-kvnu9oYg2021050708280112110000000e]
module.eks.aws_iam_instance_profile.workers[0]: Creation complete after 2s [id=test-eks-kvnu9oYg2021050708280112100000000d]
module.eks.aws_launch_configuration.workers[0]: Creating...
module.eks.aws_launch_configuration.workers[1]: Creating...
module.eks.data.http.wait_for_cluster[0]: Still reading... [10s elapsed]
module.eks.aws_launch_configuration.workers[0]: Still creating... [10s elapsed]
module.eks.aws_launch_configuration.workers[1]: Still creating... [10s elapsed]
module.eks.aws_launch_configuration.workers[1]: Creation complete after 16s [id=test-eks-kvnu9oYg-worker-group-220210507082804636500000013]
module.eks.aws_launch_configuration.workers[0]: Creation complete after 16s [id=test-eks-kvnu9oYg-worker-group-120210507082804589800000012]
module.eks.random_pet.workers[0]: Creating...
module.eks.random_pet.workers[1]: Creating...
module.eks.random_pet.workers[1]: Creation complete after 0s [id=relaxed-woodcock]
module.eks.random_pet.workers[0]: Creation complete after 0s [id=exact-locust]
module.eks.aws_autoscaling_group.workers[1]: Creating...
module.eks.aws_autoscaling_group.workers[0]: Creating...
module.eks.data.http.wait_for_cluster[0]: Still reading... [20s elapsed]
module.eks.aws_autoscaling_group.workers[0]: Still creating... [10s elapsed]
module.eks.aws_autoscaling_group.workers[1]: Still creating... [10s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [30s elapsed]
module.eks.aws_autoscaling_group.workers[1]: Still creating... [20s elapsed]
module.eks.aws_autoscaling_group.workers[0]: Still creating... [20s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [40s elapsed]
module.eks.aws_autoscaling_group.workers[0]: Still creating... [30s elapsed]
module.eks.aws_autoscaling_group.workers[1]: Still creating... [30s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [50s elapsed]
module.eks.aws_autoscaling_group.workers[1]: Creation complete after 40s [id=test-eks-kvnu9oYg-worker-group-220210507082818853700000014]
module.eks.aws_autoscaling_group.workers[0]: Still creating... [40s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [1m0s elapsed]
module.eks.aws_autoscaling_group.workers[0]: Creation complete after 45s [id=test-eks-kvnu9oYg-worker-group-120210507082818856900000015]
module.eks.data.http.wait_for_cluster[0]: Still reading... [1m10s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [1m20s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [1m30s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [1m40s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [1m50s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [2m0s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [2m10s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [2m20s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [2m30s elapsed]
module.eks.data.http.wait_for_cluster[0]: Read complete after 2m32s [id=https://xxxxxxx.xxx.xxx.eks.amazonaws.com/healthz]
data.aws_eks_cluster.cluster: Reading...

It took 2m32s for module.eks.data.http.wait_for_cluster[0] to clomplete.

Todo

• Add terraform-aws-module/terraform-prodiver-http in examples
• Document why we fork the http provider
• Publish provider into terraform registry

[barryib]

@shoekstra @daroga0002 @jlpettersson can you please help me to test this ?

[barryib]

Maybe @xavfernandez could help to review and test this ?

[barryib]

How to change this url ?

[daroga0002]

Tested IRSA example with adding some explicit kubernetes resource creation:

...........
module.eks.aws_eks_cluster.this[0]: Still creating... [9m20s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [9m30s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [9m40s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [9m50s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [10m0s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [10m10s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [10m20s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [10m30s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [10m40s elapsed]
module.eks.aws_eks_cluster.this[0]: Creation complete after 10m44s [id=test-eks-irsa]
module.eks.data.http.wait_for_cluster[0]: Reading...
module.eks.data.template_file.userdata[0]: Reading...
module.eks.aws_iam_openid_connect_provider.oidc_provider[0]: Creating...
module.iam_assumable_role_admin.data.aws_iam_policy_document.assume_role_with_oidc[0]: Reading...
module.eks.aws_iam_role.workers[0]: Creating...
module.eks.data.template_file.userdata[0]: Read complete after 0s [id=5467189cf9996685ed6585631c69c9d829c9fb16b55b2e8f02752a661d8421bc]
module.iam_assumable_role_admin.data.aws_iam_policy_document.assume_role_with_oidc[0]: Read complete after 0s [id=2170324641]
module.eks.local_file.kubeconfig[0]: Creating...
module.iam_assumable_role_admin.aws_iam_role.this[0]: Creating...
.........

I spin cluster few times and each time created with success.

[stevehipwell]

@barryib would it be possible to expose the new wait_for_cluster variable as an output along the lines of control_plane_ready?

[barryib]

@barryib would it be possible to expose the new wait_for_cluster variable as an output along the lines of control_plane_ready?

@stevehipwell What do you mean by that ? Do you want me to add a new output for data.wait_for_cluster ?

[stevehipwell]

@barryib yes that's what I mean. Although in the context of the module it's the control plane not the cluster that's ready which is why I suggested the output name be changed.

[barryib]

@barryib yes that's what I mean. Although in the context of the module it's the control plane not the cluster that's ready which is why I suggested the output name be changed.

Normally when you use cluster_id or kubeconfig outputs, you don't need bother yourself with another control_plane_ready because those outputs depend already on data.http.wait_for_cluster.

[stevehipwell]

@barryib we don't use the kubeconfig output and prefer to use aws_eks_cluster_auth to get a token. We need to know when the control plane is ready as we need to remove config (due to aws/containers-roadmap#923 not being completed yet) before the workers are started. We were using curl and now have a beta version of our internal implementation using your http datasource. An output would be a tidier implementation.

[stevehipwell]

I guess we could use cluster_id to achieve this.

[barryib]

This now shipped in v16.0.0

[nauxliu]

@barryib the Forked http module does not have a darwin_arm64 release, I can't use this module on my M1 MacBook, I've created a PR to build releases with Go 1.16. Please take a look terraform-aws-modules/terraform-provider-http#5

[rkinganduril]

I was using the wait for cluster cmd for Windows compatibility. Now that the variable is removed I cannot finish my terraform with the new http module. Upon removing the wait for cluster command in favor of the new method.

[ivialex-mcd]

I was using the wait for cluster cmd for Windows compatibility. Now that the variable is removed I cannot finish my terraform with the new http module. Upon removing the wait for cluster command in favor of the new method.

I have the same problem.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.