Prebuilt Provider Packages

[skorfmann]

When thinking about distributing L2 or L3 constructs, we'll need provider packages. Without, it's quite hard to define the dependency on used providers (e.g. Null Provider).

Current State

In order to do this at the moment by the example of aws-ecr-asset:

• Create / Publish aws-ecr-asset package
• Add a note in the Readme / post install to add the null provider to cdktf.json
• When there's a breaking change in one of the upstream packages, it'll break and the user will have dig through weird error messages

Desired State

We have prebuilt provider packages, with potentially a few predefined L2 / L3 constructs for each of these providers, which allows leaving the dependency management to proper tooling (npm et al). This would also get us closer to a workflow where there's essentially no need for a local Terrafor CLI (if the Terraform Cloud workflow is chosen).

The version of distributed packages shouldn't be tied to the original provider, but always track latest. The version would be mandated by changes to cdktf the <provider>-constructs and possibly breaking changes by the actual underlying Terraform Provider / Engine.

• The provider package gets auto-generated via cdktf get
• It tracks latest of the provider
• Users which require a specific provider version for some reason, will have to fallback to generate it manually by cdktf get

Proposed Repo Structure

To allow individiual versioning of providers, I think it makes sense to use something like the following repo structure

hashicorp/terraform-cdk

• packages/cdktf
• packages/cdktf-cli

hashicorp/terraform-cdk-aws

• packages/@cdktf/aws # metapackage to pull in cdktf / aws provider / aws constructs
• packages/@cdktf/aws-provider
• packages/@cdktf/aws-constructs

hashicorp/terraform-cdk-azure

• packages/@cdktf/azure # metapackage to pull in cdktf / azure provider / azure constructs
• packages/@cdktf/azure-provider
• packages/@cdktf/azure-constructs

Conclusion

Personally, I think that L2 / L3 constructs will be the main selling point for the Terraform CDK. Therefore I'd consider this quite important.

Thoughts on this one?

[jsteinich]

This is something I just started investigating with Terrastack. I don't really have much to show, but the idea was to run an action everyday to check for updated provider versions and auto build/publish if any new ones are found.

[skorfmann]

This is something I just started investigating with Terrastack. I don't really have much to show, but the idea was to run an action everyday to check for updated provider versions and auto build/publish if any new ones are found.

Yes, that would be totally sufficient for now 👍 I sketched some code already as well. I think at the moment, the primary question is how to do the versioning of these pre-packaged providers. I have some ideas around this, gonna write them up over the weekend.

[jsteinich]

I like trying to somehow include the actual provider version, but I agree that matching exactly is likely to be problematic.
Maybe something like XAA.YBBB.ZC where X = cdktf major version, AA = provider major version, Y = cdktf minor version, BBB = provider minor version, Z = cdktf patch version, C = provider patch version. Would need to start at 1 for all cdktf versions since semvar calls for no leading zeroes. Would 0 pad provider version numbers as needed to ensure space for future increases.

[eladb]

I am wondering if it's sufficient to just somehow have the library relay the "provider requirements" to the app & the CLI instead? Publishing a package for each provider feels like an overkill...

[skorfmann]

I am wondering if it's sufficient to just somehow have the library relay the "provider requirements" to the app & the CLI instead? Publishing a package for each provider feels like an overkill...

@anubhavmishra and I talked about this, and came to the conclusion that building a custom dependency management wouldn't be worth the effort. I don't think we'd distribute packages for all providers, but a selection of the most used ones ( for now at least).

Why I think we should do this

Distribution of custom Constructs

As mentioned above, it's not possible to build custom constructs with dependencies to used providers.

Speed

I did test building a prepackaged provider for AWS / Azure. This reduced the build time for a cdktf synth by ~ 50% (~ 8s / 4s). That's quite significant (related to #151)

Getting Started Experience

It'd be a much nicer experience if a user wouldn't have to run cdktf get to get started.

Dependency Management / Versioning

For prepackaged providers we'll have multiple upstream dependencies:

• The respective provider (e.g. AWS or Google)
• Terraform Engine (indirect via cdktf)
• cdktf

I wouldn't bother trying to reflect the individual upstream versions in a prepackaged provider package. I'd rather be opinionated and assume that most users will be fine with latest.

Both, Terraform and the popular providers are quite mature and wouldn't change that often in backwards incompatible ways. For cdktf that's certainly different. Anyway, if there would be any breaking upstream changes the packaged provider would just be bumped.

As already suggested by @jsteinich I think it would make sense to publish new minor or even just patch releases when a new provider got published. This could be run on a schedule, or we somehow subscribe to release notifications.

My assumption is that most users don't care about the specific version of a given provider. It's usually fine to just go with latest. If a user would really care about a certain version, it's still possible to use cdktf get to build your own version of that provider as today.

The benefits of this approach:

• No need to build custom logic for things which can be perfectly handled by npm et al
• Reduce possible frustration for breaking changes around generated providers
• Users can use the standard tooling to get notified about new updates
• More idiomatic usage of the entire project since it fits into the expected way of using dependencies

Terraform Cloud Workflows

There's this idea floating around to support a workflow which is fully backed by the Terraform Cloud. This means, a user wouldn't even have to have the Terraform CLI installed to use cdktf (quite similar to the AWS CDK Cloudformation workflow) That's only possible when we have prepackaged providers.

Usage Information

Having prepackaged providers would also convey a bit of usage information via npm / python package download stats

Conclusion

Since prepackaged providers would just be built via cdktf get, the effort of creating these packages is quite low from my point of view. While it'll required being opinionated about versioning, I believe the provided value would outweigh potential drawbacks.

Would love to get some feedback on this!

[eladb]

My gut feeling is that this is going to blow up in complexity very quickly and I would recommend not publishing pre-built providers. I think it shouldn't be too hard to define a protocol which allows libraries to communicate which provider versions they need to operate and have the app pick that up. The app wouldn't need to generate the provider code itself, just to configure so it will be downloaded and available to the client during runtime.

[skorfmann]

My gut feeling is that this is going to blow up in complexity very quickly and I would recommend not publishing pre-built providers.

Where do you see the complexity blowing up, could you elaborate on this a bit?

I think it shouldn't be too hard to define a protocol which allows libraries to communicate which provider versions they need to operate and have the app pick that up. The app wouldn't need to generate the provider code itself, just to configure so it will be downloaded and available to the client during runtime.

You mean something along the lines of what we have with cdktf.json?

[jsteinich]

The app wouldn't need to generate the provider code itself, just to configure so it will be downloaded and available to the client during runtime.

I'm not certain that will work on all potential languages. Seems like something that would fail with build time linking.

I agree that cdktf get is a bit of an awkward step for getting starting. So is having to use cdktf.json for typical use.

Some complexity I see comes down the road when/if there are multiple supported versions of cdktf. There could end being a matrix of packages needed. However, at some point there will probably need to be a line of what is officially supported and what is unofficial. Any provider that is officially supported will need to built with each version to ensure that nothing breaks anyways, so it doesn't seem like too much extra effort to also publish.