aws_msk_cluster: support in-place Kafka version upgrade

[arafsheikh]

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #13547

Release note for CHANGELOG:

resource/aws_msk_cluster: support in-place Kafka version upgrade [GH-13547]

Output from acceptance testing:

$ make testacc TEST=./aws TESTARGS='-run=TestAccAWSMskCluster_KafkaVersionUpgrade'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSMskCluster_KafkaVersionUpgrade -timeout 120m
=== RUN   TestAccAWSMskCluster_KafkaVersionUpgrade
=== PAUSE TestAccAWSMskCluster_KafkaVersionUpgrade
=== CONT  TestAccAWSMskCluster_KafkaVersionUpgrade
--- PASS: TestAccAWSMskCluster_KafkaVersionUpgrade (4734.43s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	4736.762s
$ make testacc TEST=./aws TESTARGS='-run=TestAccAWSMskCluster_KafkaVersionUpgradeWithConfigurationInfo'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSMskCluster_KafkaVersionUpgradeWithConfigurationInfo -timeout 120m
=== RUN   TestAccAWSMskCluster_KafkaVersionUpgradeWithConfigurationInfo
=== PAUSE TestAccAWSMskCluster_KafkaVersionUpgradeWithConfigurationInfo
=== CONT  TestAccAWSMskCluster_KafkaVersionUpgradeWithConfigurationInfo
--- PASS: TestAccAWSMskCluster_KafkaVersionUpgradeWithConfigurationInfo (6143.62s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	6145.800s
$ make testacc TEST=./aws TESTARGS='-run=TestAccAWSMskCluster_KafkaVersionDowngrade'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSMskCluster_KafkaVersionDowngrade -timeout 120m
=== RUN   TestAccAWSMskCluster_KafkaVersionDowngrade
=== PAUSE TestAccAWSMskCluster_KafkaVersionDowngrade
=== CONT  TestAccAWSMskCluster_KafkaVersionDowngrade
--- PASS: TestAccAWSMskCluster_KafkaVersionDowngrade (3230.12s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	3232.396s

[silvestriluca]

Hello.

Is there an ETA on this PR ? It's stale from quite few days and it's a very useful addition for MSK users.

[cdenneen]

Need to perform in-place upgrade and the feature has been implemented almost a year ago now. Can this PR be merged?

[aheuermann]

Could someone please take a look at this PR? Would love to be able to do rolling kafka upgrades. Right now it requires a full replacement.

[arafsheikh]

Hi @bflad, this would be a valuable feature for a lot of us. Happy to discuss any concerns with the current solution.

[bill-rich]

This looks really great overall. Thank you for taking the time to write this. I had one question/request about downgrading, and one for test coverage.

[anGie44]

great work @arafsheikh!

output of acceptance tests (passing minus known failure tracked in #16249):

--- PASS: TestAccAWSMskCluster_basic (1596.80s)
--- PASS: TestAccAWSMskCluster_EncryptionInfo_EncryptionAtRestKmsKeyArn (1599.19s)
--- PASS: TestAccAWSMskClusterDataSource_Name (1603.31s)
--- PASS: TestAccAWSMskCluster_EnhancedMonitoring (1682.11s)
--- PASS: TestAccAWSMskCluster_EncryptionInfo_EncryptionInTransit_InCluster (1724.10s)
--- PASS: TestAccAWSMskCluster_EncryptionInfo_EncryptionInTransit_ClientBroker (1724.86s)
--- PASS: TestAccAWSMskCluster_Tags (1755.12s)
--- PASS: TestAccAWSMskCluster_OpenMonitoring (1902.44s)
--- PASS: TestAccAWSMskCluster_BrokerNodeGroupInfo_EbsVolumeSize (1981.06s)
--- PASS: TestAccAWSMskCluster_NumberOfBrokerNodes (2274.10s)
--- PASS: TestAccAWSMskCluster_ConfigurationInfo_Revision (2377.73s)
--- PASS: TestAccAWSMskCluster_KafkaVersionDowngrade (3449.64s)
--- PASS: TestAccAWSMskCluster_KafkaVersionUpgrade (5527.38s)
--- PASS: TestAccAWSMskCluster_KafkaVersionUpgradeWithConfigurationInfo (5535.80s)
--- FAIL: TestAccAWSMskCluster_LoggingInfo (1702.11s) -- ongoing test failure

[ghost]

This has been released in version 3.16.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!