V3 Packaging Tools Strips out nested deployments

[seanstark]

Summary

When a CCF (Codeless Connector Framework) solution uses a customized StorageAccountBlobContainer data connector, re-running the V3 packaging tool (createSolutionV3.ps1) silently discards the customizations in the generated mainTemplate.json. The storage-connector ARM is produced from hardcoded PowerShell, not from the solution's source connector files, so any author customization that lives only in the generated output is lost on the next repackage.

Example Solution

https://github.com/seanstark/Azure-Sentinel/tree/seanstark-cloudflare/Solutions/Cloudflare%20CCF

Root Cause

The CCF pipeline generates the StorageAccountBlobContainer connector from fixed code rather than source:

• Tools/Create-Azure-Sentinel-Solution/common/createCCPConnector.ps1 dispatches storageaccountblobcontainer to CreateStorageAccountBlobContainerResourceProperties.
• Tools/Create-Azure-Sentinel-Solution/common/storageAccountDeploymentTemplate.ps1 emits a fixed template via Get-StorageAccountDeploymentTemplate and Set-ResourceVariables.

Because the generator owns the storage template, there is no source-of-truth file where connector customizations can be expressed and preserved.

Impact — What Gets Destroyed on Repackage

• Multi-collector ContextPane (multiple account/storage/container/topic/table fields) is replaced by a single blobContainerUri parameter, with names parsed from it.
• Account-prefixed unique resource names (so multiple collectors coexist) are replaced by a fixed nestedDeploymentName = "CreateDataFlowResources", causing name collisions.
• deploymentScripts RBAC-propagation wait (e.g. Start-Sleep 60) is not emitted at all.
• Custom table-name parameter → effective table name is replaced by a hardcoded table.
• System-topic location via reference(storageAccountId).location is replaced by parameters('StorageAccountLocation').
• forceUpdateTag (force re-run on each "Add new collector") is dropped.

Additionally, the generator's ARM-expression escaping blindly prepends [ to produce [[. It is not idempotent, so already-escaped ([[) expressions in any passthrough/custom content get mangled.

Steps to Reproduce

1. Author a CCF solution with a customized StorageAccountBlobContainer connector (multi-collector UI, Event Grid wiring, queue/blob RBAC role assignments, nested deployment, and a deploymentScripts RBAC-wait).
2. Run createSolutionV3.ps1 to package the solution.
3. Inspect the generated Package/mainTemplate.json.

Expected vs. Actual

• Expected: Author customizations to the storage connector are preserved across repackaging.
• Actual: The connector is regenerated from the canned template; Event Grid wiring, per-collector unique naming, RBAC role assignments, the deploymentScripts wait, custom table name, and forceUpdateTag are dropped or overwritten.

Proposed Fixes (in priority order)

1. Custom-resources sidecar file (best general fix). Allow an optional file in the CCF connector folder (e.g. *_DeploymentResources.json) containing resources / variables / parameters. After building the definition + poller + DCR + table, the generator appends these verbatim into the connections mainTemplate, so Event Grid, RBAC, nested deployment, and the deploymentScripts wait regenerate correctly every time.
2. Idempotent ARM-expression escaping. Make the [ -> [[ escaping skip expressions already prefixed with [[, so passthrough resources are not corrupted.
3. Externalize the storage template (targeted). Move the default Get-StorageAccountDeploymentTemplate / Set-ResourceVariables content into a template file; if the connector folder supplies its own storage template, use it instead of the built-in.
4. Per-connector "raw passthrough" flag. A Solution_*.json flag (e.g. "DataConnectorsPassthrough") that tells V3 to wrap a fully-authored connector ARM template in the contentTemplate/metadata envelope as-is, skipping lossy re-derivation.

Recommended: implement #1 + #2 (general fix + corruption fix), optionally #4 for fully-authored connectors.

Affected Files

• Tools/Create-Azure-Sentinel-Solution/common/createCCPConnector.ps1
• Tools/Create-Azure-Sentinel-Solution/common/storageAccountDeploymentTemplate.ps1