[azdatalake][STG84] Append with flush (#22245)

* Update package versions (#22201)

* Update package versions

* upgrade to MSAL v1.2.1

* add token_type to fake tokens

* revert debugging

---------

Co-authored-by: Charles Lowell <10964656+chlowell@users.noreply.github.com>

* Increment package version after release of messaging/azeventgrid (#22060)

* Increment package version after release of ai/azopenai (#22126)

* azblob: STG91 upgrade (#22194)

* azfile: STG91 upgrade (#22193)

* update source link to new azopenai location (#22208)

* link current instead of depricated path

* remove newline at end of file

* More azappconfig cleanup (#22209)

* More azappconfig cleanup

Update BeginCreateSnapshot to return the complete Snapshot type.
Remove unused response types.
Added some missed error checking in a few tests.

* fix a few doc comments

* Fix spelling (preceeding=> preceding) (#22202)

Co-authored-by: Anton Kolesnyk <41349689+antkmsft@users.noreply.github.com>

* Storage: Reject HTTP endpoints (#22183)

* [azeventhubs] Fixing an issue where a processor wouldn't grab partitions (#22153)

The processor load balancer had a bug where a processor would not claim partitions even if it didn't yet have the proper share.

Added in stress tests (balance, multibalance) to run several scenarios with our supported strategies and different parallel owners.

Fixes #22097

* Clean up ETags in azappconfig (#22212)

Use azcore.ETag instead of string for public surface area.
Add doc comments for non-standard ETag option names.

* [azopenai] Fixing issue where you can't use whisper with m4a files. (#22210)

Fixing issue where you can't use whisper with m4a files.

* It's one of the formats that doesn't seem to be recognized without an explicit file extension, which you can pass via Filename
* My tests were too heavily dependent on implementation details of the models. Changing this out to check that things are working correctly without checking the exact contents of the response.
* Also, rerecorded tests since we're doing multiple audio tests as well.

Fixes #22195

* Updating changelog for STG91 preview (#22216)

* azfile: STG 91 Preview (#22217)

* Increment package version after release of storage/azblob (#22219)

* [azopenai] Update changelog for release. (#22215)

* Replace string with azcore.ETag (#22218)

Missed one in last round of clean-up.

* Add script to go mod tidy modules (#22214)

Will tidy all modules under the current working directory or under the
specified directory.

* Increment package version after release of storage/azfile (#22220)

* Added Simon and Kushagra as code owners on GO Cosmos SDK (#22221)

* [azservicebus] Updating batching to allow for a configurable wait time (#22154)

Updating batching to allow for a configurable wait time. Can lead to fuller batches for people that want to tune it.

Fixes #19172

* Sync eng/common directory with azure-sdk-tools for PR 7459 (#22222)

* check for the presence of a compatible powershell. ensure that we always return a list of tags

* allow the script to require pshell6+

* remove the en-us from the link

---------

Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>

* Fix image tag for multiimage deployments (#22224)

Co-authored-by: Liudmila Molkova <limolkova@microsoft.com>

* Refresh azcontainerregistry module (#22161)

* upgrade to latest codegen

* merge resolver

* enable span and add changelog

* update module name

* fix generator tool (#22205)

* fix generator tool

* gofmt

* fix

* goimports

* azdatalake preview stg82

* azdatalake preview stg82 (#22227)

* Rename azidentity/cache pipeline (#22231)

* Handle errors in mod tidy script (#22234)

* Sync eng/common directory with azure-sdk-tools for PR 7445 (#22233)

* Add package filter parameter to select recursively found packages in release

* Handle empty value for packageFilter

---------

Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>

* Use package filter to prevent recursive package publish (#22232)

* append with flush

---------

Co-authored-by: Rick Winter <rick.winter@microsoft.com>
Co-authored-by: Charles Lowell <10964656+chlowell@users.noreply.github.com>
Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com>
Co-authored-by: Sourav Gupta <98318303+souravgupta-msft@users.noreply.github.com>
Co-authored-by: Felix <9039899+fkarg@users.noreply.github.com>
Co-authored-by: Joel Hendrix <jhendrix@microsoft.com>
Co-authored-by: Anton Kolesnyk <41349689+antkmsft@users.noreply.github.com>
Co-authored-by: Richard Park <51494936+richardpark-msft@users.noreply.github.com>
Co-authored-by: Kushagra Thapar <kuthapar@microsoft.com>
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
Co-authored-by: Liudmila Molkova <limolkova@microsoft.com>
Co-authored-by: Chenjie Shi <tadelesh.shi@live.cn>
Co-authored-by: Peng Jiahui <46921893+Alancere@users.noreply.github.com>
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>

Author: 15 people

.github/CODEOWNERS

@@ -28,7 +28,7 @@
 /sdk/data/azappconfig/        @antkmsft @jhendrixMSFT @rickwinter
 
 # PRLabel: %Cosmos
-/sdk/data/azcosmos/           @ealsur @kirankumarkolli
+/sdk/data/azcosmos/           @ealsur @kirankumarkolli @simorenoh @kushagraThapar
 
 # PRLabel: %Tables
 /sdk/data/aztables/           @jhendrixMSFT

eng/common/pipelines/templates/steps/create-tags-and-git-release.yml

@@ -1,6 +1,7 @@
 parameters:
   ArtifactLocation: 'not-specified'
   PackageRepository: 'not-specified'
+  PackageFilter: ''
   ReleaseSha: 'not-specified'
   RepoId: $(Build.Repository.Name)
   WorkingDirectory: ''
@@ -14,10 +15,11 @@ steps:
     arguments: >
       -artifactLocation ${{ parameters.ArtifactLocation }}
       -packageRepository ${{ parameters.PackageRepository }}
+      -packageFilter "${{ parameters.PackageFilter }}"
       -releaseSha ${{ parameters.ReleaseSha }}
       -repoId ${{ parameters.RepoId }}
       -workingDirectory '${{ parameters.WorkingDirectory }}'
     pwsh: true
   timeoutInMinutes: 5
   env:
-    GH_TOKEN: $(azuresdk-github-pat)
+    GH_TOKEN: $(azuresdk-github-pat)

eng/common/scripts/Submit-PullRequest.ps1

@@ -23,12 +23,12 @@ The body message for the pull request.
 The labels added to the PRs. Multple labels seperated by comma, e.g "bug, service"
 .PARAMETER UserReviewers
 User reviewers to request after opening the PR. Users should be a comma-
-separated list with no preceeding `@` symbol (e.g. "user1,usertwo,user3")
+separated list with no preceding `@` symbol (e.g. "user1,usertwo,user3")
 .PARAMETER TeamReviewers
 List of github teams to add as reviewers
 .PARAMETER Assignees
 Users to assign to the PR after opening. Users should be a comma-separated list
-with no preceeding `@` symbol (e.g. "user1,usertwo,user3")
+with no preceding `@` symbol (e.g. "user1,usertwo,user3")
 .PARAMETER CloseAfterOpenForTesting
 Close the PR after opening to save on CI resources and prevent alerts to code
 owners, assignees, requested reviewers, or others.

eng/common/scripts/create-tags-and-git-release.ps1

@@ -7,6 +7,7 @@ param (
   $artifactLocation, # the root of the artifact folder. DevOps $(System.ArtifactsDirectory)
   $workingDirectory, # directory that package artifacts will be extracted into for examination (if necessary)
   $packageRepository, # used to indicate destination against which we will check the existing version.
+  $packageFilter,
   # valid options: PyPI, Nuget, NPM, Maven, C, CPP
   # used by CreateTags
   $releaseSha, # the SHA for the artifacts. DevOps: $(Release.Artifacts.<artifactAlias>.SourceVersion) or $(Build.SourceVersion)
@@ -26,6 +27,11 @@ Write-Host "Using API URL $apiUrl"
 # VERIFY PACKAGES
 $pkgList = VerifyPackages -artifactLocation $artifactLocation -workingDirectory $workingDirectory -apiUrl $apiUrl -releaseSha $releaseSha -continueOnError $continueOnError
 
+if ($packageFilter) {
+    Write-Host "Filtering discovered packages to '$packageFilter'"
+    [array]$pkgList = $pkgList | Where-Object { $_.PackageId -like $packageFilter }
+}
+
 if ($pkgList) {
   Write-Host "Given the visible artifacts, github releases will be created for the following:"
 

eng/common/scripts/stress-testing/stress-test-deployment-lib.ps1

@@ -257,7 +257,7 @@ function DeployStressPackage(
     if ($pkg.Dockerfile -or $pkg.DockerBuildDir) {
         throw "The chart.yaml docker config is deprecated, please use the scenarios matrix instead."
     }
-    
+
 
     foreach ($dockerBuildConfig in $dockerBuildConfigs) {
         $dockerFilePath = $dockerBuildConfig.dockerFilePath
@@ -285,7 +285,7 @@ function DeployStressPackage(
             $dockerBuildCmd += $dockerBuildFolder
 
             Run @dockerBuildCmd
-            
+
             Write-Host "`nContainer image '$imageTag' successfully built. To run commands on the container locally:" -ForegroundColor Blue
             Write-Host "  docker run -it $imageTag" -ForegroundColor DarkBlue
             Write-Host "  docker run -it $imageTag <shell, e.g. 'bash' 'pwsh' 'sh'>" -ForegroundColor DarkBlue
@@ -302,7 +302,7 @@ function DeployStressPackage(
             }
         }
         $generatedHelmValues.scenarios = @( foreach ($scenario in $generatedHelmValues.scenarios) {
-            $dockerPath = if ("image" -notin $scenario) {
+            $dockerPath = if ("image" -notin $scenario.keys) {
                 $dockerFilePath
             } else {
                 Join-Path $pkg.Directory $scenario.image
@@ -476,7 +476,7 @@ function generateRetryTestsHelmValues ($pkg, $releaseName, $generatedHelmValues)
             $failedJobsScenario += $job.split("-$($pkg.ReleaseName)")[0]
         }
     }
-    
+
     $releaseName = "$($pkg.ReleaseName)-$revision-retry"
 
     $retryTestsHelmVal = @{"scenarios"=@()}

eng/common/testproxy/scripts/tag-merge/README.md

@@ -11,6 +11,7 @@ This script merely allows the abstraction of some of this "combination" work.
 
 ### PreReqs
 
+- Must have []`pshell 6+`](https://learn.microsoft.com/powershell/scripting/install/installing-powershell-on-windows)
 - Must have `git` available on your PATH
 - Must have the `test-proxy` available on your PATH
   - `test-proxy` is honored when the proxy is installed as a `dotnet tool`

eng/common/testproxy/scripts/tag-merge/merge-proxy-tags.ps1

@@ -1,3 +1,6 @@
+#Requires -Version 6.0
+#Requires -PSEdition Core
+
 <#
 .SYNOPSIS
 Merge multiple asset tagss worth of content into a single asset tag.
@@ -66,9 +69,6 @@ function Git-Command($CommandString, $WorkingDirectory, $HardExit=$true) {
 }
 
 function Resolve-Proxy {
-    # this script requires the presence of git
-    Test-Exe-In-Path -ExeToLookFor "git" | Out-Null
-
     $testProxyExe = "test-proxy"
     # this script requires the presence of the test-proxy on the PATH
     $proxyToolPresent = Test-Exe-In-Path -ExeToLookFor "test-proxy" -ExitOnError $false
@@ -184,7 +184,13 @@ function Start-Message($AssetsJson, $TargetTags, $AssetsRepoLocation, $MountDire
 function Finish-Message($AssetsJson, $TargetTags, $AssetsRepoLocation, $MountDirectory) {
     $len = $TargetTags.Length
 
-    Write-Host "`nSuccessfully combined $len tags. Invoke `"test-proxy push " -NoNewLine
+    if ($TargetTags.GetType().Name -eq "String") {
+        $len = 1
+    }
+
+    $suffix = if ($len -gt 1) { "s" } else { "" }
+
+    Write-Host "`nSuccessfully combined $len tag$suffix. Invoke `"test-proxy push " -NoNewLine
     Write-Host $AssetsJson -ForegroundColor Green -NoNewLine
     Write-Host "`" to push the results as a new tag."
 }
@@ -270,12 +276,22 @@ function Combine-Tags($RemainingTags, $AssetsRepoLocation, $MountDirectory){
     # if we have successfully gotten to the end without any non-zero exit codes...delete the mergeprogress file, we're g2g
     Cleanup-Incomplete-Progress $MountDirectory
 
-    return $pushedTags
+    return @($pushedTags)
 }
 
 $ErrorActionPreference = "Stop"
 
-# resolve the proxy location so that we can invoke it easily
+# this script requires the presence of git
+Test-Exe-In-Path -ExeToLookFor "git" | Out-Null
+
+# this script expects at least powershell 6 (core)
+
+if ($PSVersionTable["PSVersion"].Major -lt 6) {
+    Write-Error "This script requires a version of powershell newer than 6. See http://aka.ms/powershell for resolution."
+    exit 1
+}
+
+# resolve the proxy location so that we can invoke it easily, if not present we exit here.
 $proxyExe = Resolve-Proxy
 
 $AssetsJson = Resolve-Path $AssetsJson

eng/pipelines/templates/jobs/archetype-go-release.yml

@@ -74,10 +74,11 @@ stages:
                 - template: /eng/common/pipelines/templates/steps/create-tags-and-git-release.yml
                   parameters:
                     ArtifactLocation: $(Build.SourcesDirectory)/sdk/${{ parameters.ServiceDirectory }}
+                    PackageFilter: sdk/${{ parameters.ServiceDirectory }}
                     ReleaseSha: $(Build.SourceVersion)
                     RepoId: Azure/azure-sdk-for-go
                     WorkingDirectory: $(System.DefaultWorkingDirectory)
-      
+
       - ${{ if not(and(startsWith(parameters.ServiceDirectory, 'resourcemanager'), ne(parameters.ServiceDirectory, 'resourcemanager/internal'))) }}:
         - deployment: UpdatePackageVersion
           displayName: "Update Package Version"

eng/scripts/modtidy.ps1

@@ -0,0 +1,23 @@
+Param(
+    [string]$searchPath = $PWD.Path
+)
+
+$modFiles = Get-ChildItem -Path $searchPath -Include "go.mod" -Recurse
+
+$tidyErrors = $modFiles | ForEach-Object -Parallel {
+    Set-Location $_.Directory
+    Write-Host (Get-Location)
+    $output = go mod tidy 2>&1
+    if ($LASTEXITCODE) {
+        return @{ Directory = $_.Directory; Output = $output }
+    }
+}
+
+if ($tidyErrors) {
+    Write-Error "Encountered the following tidy failures:" -ErrorAction 'Continue'
+    foreach ($err in $tidyErrors) {
+        Write-Host "=== $($err.Directory) ==="
+        $err.Output
+    }
+    exit 1
+}

eng/tools/generator/cmd/v2/common/fileProcessor.go

@@ -5,6 +5,7 @@ package common
 
 import (
 	"bytes"
+	"errors"
 	"fmt"
 	"io/fs"
 	"io/ioutil"
@@ -543,11 +544,9 @@ func replaceModuleImport(path, rpName, namespaceName, previousVersion, currentVe
 		if err != nil {
 			return err
 		}
-
 		if info.IsDir() {
 			return nil
 		}
-
 		suffix := false
 		for i := 0; i < len(suffixes) && !suffix; i++ {
 			suffix = strings.HasSuffix(info.Name(), suffixes[i])
@@ -566,11 +565,25 @@ func replaceModuleImport(path, rpName, namespaceName, previousVersion, currentVe
 				}
 			}
 		}
-
 		return nil
 	})
 }
 
+func getModuleVersion(autorestPath string) (*semver.Version, error) {
+	data, err := os.ReadFile(autorestPath)
+	if err != nil {
+		return nil, err
+	}
+
+	for _, line := range strings.Split(string(data), "\n") {
+		if strings.HasPrefix(line, autorest_md_module_version_prefix) {
+			return semver.NewVersion(strings.TrimSpace(line[len(autorest_md_module_version_prefix):]))
+		}
+	}
+
+	return nil, errors.New("module-version does not exist in autorest.md")
+}
+
 func existSuffixFile(path, suffix string) bool {
 
 	existed := false