Merge branch 'main' of https://github.com/Azure/azure-sdk-for-python into network_ga

* 'main' of https://github.com/Azure/azure-sdk-for-python: (63 commits)
  check only once whether artifact extension exists (Azure#29594)
  code and test (Azure#29530)
  fix: support data binding expression for resources.xxx (Azure#29559)
  code and test (Azure#29558)
  [ServiceBus] enable sovereign cloud testing (Azure#29202)
  [AutoRelease] t2-communication-2023-03-20-22792(can only be merged by SDK owner) (Azure#29453)
  [Identity] Allow configurable process timeouts (Azure#28290)
  Increment package version after release of azure-containerregistry (Azure#29582)
  Update optional (Azure#28435)
  [Monitor][Query] Add sample notebook for large queries (Azure#28148)
  Update CHANGELOG.md (Azure#29580)
  [ServiceBus] remove msrest (Azure#29324)
  Explicitly set PSNativeCommandArgumentPassing to Legacy for git push script (Azure#29554)
  [ML][Pipelines] Support queue settings in pipeline job (Azure#29209)
  add experimental decorator for datatransfer (Azure#29562)
  Featurestore/online store crud new pr (Azure#29546)
  switch tooling over to typespec (Azure#29501)
  set up base path for feature set in load function (Azure#29544)
  Address credential description in docstring (Azure#29549)
  [Monitor] Regen query and ingestion (Azure#29504)
  ...

Author: iscai-msft

.github/CODEOWNERS

@@ -917,7 +917,7 @@
 #/<NotInRepo>/          @ccmshowbackdevs
 
 # ServiceLabel: %Monitor - Exporter %Service Attention
-#/<NotInRepo>/          @@cijothomas @@reyang @@rajkumar-rangaraj @@TimothyMothra @@vishweshbankwar @@ramthi
+#/<NotInRepo>/          @lzchen @jeremydvoss
 
 
 # Management Plane

.gitignore

@@ -53,6 +53,7 @@ _azci_*.log
 .artifacts
 .tmp_whl_dir
 .logs
+.proxy
 
 # tox environment folders
 .tox/
@@ -147,4 +148,4 @@ sdk/cosmos/azure-cosmos/test/test_config.py
 *_python.json
 
 # temporary folder to refresh SDK with cadl
-TempCadlFiles/
+TempTypeSpecFiles/

.vscode/cspell.json

@@ -392,6 +392,25 @@
     "whls"
   ],
   "overrides": [
+    {
+      "filename": "doc/dev/test_proxy_migration_guide.md",
+      "words": [
+        "pytestmarkparametrize"
+      ]
+    },
+    {
+      "filename": "tools/azure-sdk-tools/devtools_testutils/proxy_startup.py",
+      "words": [
+        "certifi",
+        "passenv"
+      ]
+    },
+    {
+      "filename": "tools/azure-sdk-tools/tests/integration/test_proxy_startup.py",
+      "words": [
+        "spinup"
+      ]
+    },
     {
       "filename": "sdk/remoterendering/**",
       "words": [
@@ -603,7 +622,8 @@
         "ierr",
         "mymodel",
         "heif",
-        "racwd"
+        "racwd",
+        "rrggbb"
       ]
     },
     {

doc/dev/issues/resolve_issues_effectively.md

@@ -11,7 +11,7 @@ Most issues can be classified into 3 categories. They are
 ## Usage error (For users)
 If you are not familiar with the SDK usage of a service, you can find relevant examples in [this repo][sample repo] in most cases.
 
-For some common errors, you can check [here](#Summary-Of-Common-Errors).
+For some common errors, you can check [here](#summary-of-common-errors).
 
 ## Feature Request(For users)
 

doc/dev/static_type_checking_cheat_sheet.md

@@ -164,6 +164,24 @@ def foo(
     ...
 ```
 
+- Do use `Optional` if a parameter can be typed as `Any` or `None`. `Optional[Any]`, or `Union[Any, None]`, is __not__ equal to `Any`.
+
+```python
+from typing import Optional, Any
+
+# Yes
+def foo(
+    bar: Optional[Any] = None,
+) -> None:
+    bar.append(1)  # error caught at type checking time: Item "None" of "Optional[Any]" has no attribute "append"
+
+# No
+def foo(
+    bar: Any = None,
+) -> None:
+    bar.append(1)  # error caught at runtime: AttributeError: 'NoneType' object has no attribute 'append'
+```
+
 ### Collections
 
 - Do familiarize yourself with the supported operations of various abstract collections in the [collections.abc](https://docs.python.org/3/library/collections.abc.html#collections-abstract-base-classes) docs.

doc/dev/test_proxy_migration_guide.md

@@ -24,7 +24,7 @@ Please refer to the [troubleshooting guide][troubleshooting] if you have any iss
 - [Advanced details](#advanced-details)
   - [What does the test proxy do?](#what-does-the-test-proxy-do)
   - [How does the test proxy know when and what to record or play back?](#how-does-the-test-proxy-know-when-and-what-to-record-or-play-back)
-  - [Start the proxy manually](#start-the-proxy-manually)
+  - [Use pytest.mark.parametrize with migrated tests](#use-pytestmarkparametrize-with-migrated-tests)
 
 ## Update existing tests
 
@@ -96,10 +96,7 @@ Resource preparers need a management client to function, so test classes that us
 
 ### Perform one-time setup
 
-1. Docker (or Podman) is a requirement for using the test proxy. You can install Docker from [docs.docker.com][docker_install], or install Podman at [podman.io][podman]. To use Podman, set an alias for `podman` to replace the `docker` command.
-2. After installing, make sure Docker/Podman is running and is using Linux containers before running tests.
-3. Follow the instructions [here][proxy_cert_docs] to complete setup. You need to trust a certificate on your machine in
-order to communicate with the test proxy over a secure connection.
+The test proxy uses a self-signed certificate to communicate with HTTPS. Follow the general setup instructions [here][proxy_cert_docs] to trust this certificate locally.
 
 ### Start the proxy server
 
@@ -120,8 +117,8 @@ def start_proxy(test_proxy):
     return
 ```
 
-The `test_proxy` fixture will fetch the test proxy Docker image and create a new container called
-`ambitious_azsdk_test_proxy`, which will be deleted after test execution unless interrupted.
+The `test_proxy` fixture will download a test proxy executable if one isn't available locally, start the tool, and stop
+it after tests complete.
 
 If your tests already use an `autouse`d, session-level fixture for tests, you can accept the `test_proxy` parameter in
 that existing fixture instead of adding a new one. For an example, see the [Register sanitizers](#register-sanitizers)
@@ -141,6 +138,9 @@ Recordings for a given package will end up in that package's `/tests/recordings`
 do. Recordings that use the test proxy are `.json` files instead of `.yml` files, so migrated test suites no longer
 need old `.yml` recordings.
 
+After migrating to use the test proxy, libraries can and are encouraged to use out-of-repo recordings. For more
+information, refer to the [recording migration guide][recording_migration].
+
 > **Note:** support for configuring live or playback tests with a `testsettings_local.cfg` file has been
 > deprecated in favor of using just `AZURE_TEST_RUN_LIVE`.
 
@@ -153,7 +153,7 @@ Instead, sanitizers (as well as matchers and transforms) can be registered on th
 `add_general_string_sanitizer`. Other sanitizers are available for more specific scenarios and can be found at
 [devtools_testutils/sanitizers.py][py_sanitizers].
 
-Sanitizers, matchers, and transforms remain registered until the proxy container is stopped, so for any sanitizers that
+Sanitizers, matchers, and transforms remain registered until the proxy tool is stopped, so for any sanitizers that
 are shared by different tests, using a session fixture declared in a `conftest.py` file is recommended. Please refer to
 [pytest's scoped fixture documentation][pytest_fixtures] for more details.
 
@@ -403,36 +403,6 @@ Running tests in playback follows the same pattern, except that requests will be
 The `recorded_by_proxy` and `recorded_by_proxy_async` decorators send the appropriate requests at the start and end of
 each test case.
 
-### Start the proxy manually
-
-There are two options for manually starting and stopping the test proxy: one uses a PowerShell command, and one uses
-methods from `devtools_testutils`.
-
-#### PowerShell
-
-There is a [PowerShell script][docker_start_proxy] in `eng/common/testproxy` that will fetch the proxy Docker image if
-you don't already have it, and will start or stop a container running the image for you. You can run the following
-command from the root of the `azure-sdk-for-python` directory to start the container whenever you want to make the test
-proxy available for running tests:
-
-```powershell
-.\eng\common\testproxy\docker-start-proxy.ps1 "start"
-```
-
-Note that the proxy is available as long as the container is running. In other words, you don't need to start and
-stop the container for each test run or between tests for different SDKs. You can run the above command in the morning
-and just stop the container whenever you'd like. To stop the container, run the same command but with `"stop"` in place
-of `"start"`.
-
-#### Python
-
-There are two methods in `devtools_testutils`, [start_test_proxy][start_test_proxy] and
-[stop_test_proxy][stop_test_proxy], that can be used to manually start and stop the test proxy. Like
-`docker-start-proxy.ps1`, `start_test_proxy` will automatically fetch the proxy Docker image for you and start the
-container if it's not already running.
-
-For more details on proxy startup, please refer to the [proxy documentation][detailed_docs].
-
 ### Use `pytest.mark.parametrize` with migrated tests
 
 Migrating tests to use basic `pytest` tools allows us to take advantage of helpful features such as
@@ -493,12 +463,10 @@ client to the test.
 
 
 [detailed_docs]: https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md
-[docker_install]: https://docs.docker.com/get-docker/
-[docker_start_proxy]: https://github.com/Azure/azure-sdk-for-python/blob/main/eng/common/testproxy/docker-start-proxy.ps1
 
 [env_var_loader]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/envvariable_loader.py
 
-[general_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/README.md
+[general_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md
 
 [mgmt_recorded_test_case]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/mgmt_recorded_testcase.py
 
@@ -507,7 +475,6 @@ client to the test.
 [parametrize_class]: https://github.com/Azure/azure-sdk-for-python/blob/d92b63b9976b0025b274016c49a250fb7c4d7333/sdk/keyvault/azure-keyvault-keys/tests/_test_case.py#L59
 [pipelines_ci]: https://github.com/Azure/azure-sdk-for-python/blob/5ba894966ed6b0e1ee8d854871f8c2da36a73d79/sdk/eventgrid/ci.yml#L30
 [pipelines_live]: https://github.com/Azure/azure-sdk-for-python/blob/e2b5852deaef04752c1323d2ab0958f83b98858f/sdk/textanalytics/tests.yml#L26-L27
-[podman]: https://podman.io/
 [proxy_cert_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/documentation/test-proxy/trusting-cert-per-language.md
 [py_sanitizers]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/sanitizers.py
 [pytest_collection]: https://docs.pytest.org/latest/goodpractices.html#test-discovery

doc/dev/test_proxy_troubleshooting.md

@@ -10,23 +10,15 @@ GitHub repository, and documentation of how to set up and use the proxy can be f
 ## Table of contents
 - [Guide for test proxy troubleshooting](#guide-for-test-proxy-troubleshooting)
     - [Table of contents](#table-of-contents)
-    - [General troubleshooting tip](#general-troubleshooting-tip)
     - [Test collection failure](#test-collection-failure)
     - [Errors in tests using resource preparers](#errors-in-tests-using-resource-preparers)
     - [Playback failures from body matching errors](#playback-failures-from-body-matching-errors)
     - [Recordings not being produced](#recordings-not-being-produced)
-    - [KeyError during container startup](#keyerror-during-container-startup)
-    - [ConnectionError during test startup](#connectionerror-during-test-startup)
+    - [ConnectionError during tests](#connectionerror-during-tests)
     - [Different error than expected when using proxy](#different-error-than-expected-when-using-proxy)
     - [Test setup failure in test pipeline](#test-setup-failure-in-test-pipeline)
     - [Fixture not found error](#fixture-not-found-error)
 
-## General troubleshooting tip
-
-For any issue that may come up, it's generally a good idea to first try deleting any existing proxy container (which
-will be called `ambitious_azsdk_test_proxy`) and creating a new one by running tests. This will fetch the latest tag of
-the test proxy Docker container, meaning the latest version of the proxy tool will be used.
-
 ## Test collection failure
 
 Because tests are now using pure `pytest` conventions without `unittest.TestCase` components, discovering tests with
@@ -56,22 +48,14 @@ matching enabled by default.
 
 ## Recordings not being produced
 
-First, make sure that the environment variable `AZURE_SKIP_LIVE_RECORDING` isn't set to "true". If it's not and live
-tests still aren't producing recordings, try deleting the `ambitious_azsdk_test_proxy` Docker container and re-running
-tests. The recording storage location is determined when the test proxy Docker container is created. If there are
-multiple local copies of the `azure-sdk-for-python` repo on your machine, the container could be storing recordings in
-the wrong repo.
-
-## KeyError during container startup
-
-Try updating your machine's version of Docker. Older versions of Docker may not return a status to indicate whether or
-not the proxy container is running, which the [proxy_startup.py][proxy_startup] script needs to determine.
+Ensure the environment variable `AZURE_SKIP_LIVE_RECORDING` **isn't** set to "true", and that `AZURE_TEST_RUN_LIVE`
+**is** set to "true".
 
-## ConnectionError during test startup
+## ConnectionError during tests
 
-For example, you may see a `requests.exceptions.ConnectionError` when trying to contact URL `/Info/Available`. This
-means that the test proxy tool wasn't started up properly, so requests to the tool are failing. Make sure Docker is
-installed and is up to date, and ensure that Linux containers are being used.
+For example, you may see a `requests.exceptions.ConnectionError` when trying to make service or sanitizer setup
+requests. This means that the test proxy tool never started correctly; ensure the `test_proxy` fixture is being invoked
+during test startup so that the tool is available during tests.
 
 ## Different error than expected when using proxy
 

doc/dev/tests.md

@@ -169,10 +169,7 @@ To migrate an existing test suite to use the test proxy, or to learn more about
 
 ### Perform one-time test proxy setup
 
-1. Docker (or Podman) is a requirement for using the test proxy. You can install Docker from [docs.docker.com][docker_install], or install Podman at [podman.io][podman]. To use Podman, set an alias for `podman` to replace the `docker` command.
-2. After installing, make sure Docker/Podman is running and is using Linux containers before running tests.
-3. Follow the instructions [here][proxy_cert_docs] to complete setup. You need to trust a certificate on your machine in
-order to communicate with the test proxy over a secure connection.
+The test proxy uses a self-signed certificate to communicate with HTTPS. Follow the general setup instructions [here][proxy_cert_docs] to trust this certificate locally.
 
 ### Set up test resources
 
@@ -383,11 +380,8 @@ to the process of updating recordings.
 
 - The targeted library is already migrated to use the test proxy.
 - Git version > 2.25.0 is to on the machine and in the path. Git is used by the script and test proxy.
-- [Docker][docker_install] or [Podman][podman] is installed.
 - Global [git config settings][git_setup] are configured for `user.name` and `user.email`.
   - These settings are also set with environment variables `GIT_COMMIT_OWNER` and `GIT_COMMIT_EMAIL`, respectively (in your environment or your local `.env` file).
-- The environment variable `GIT_TOKEN` is set to a valid [personal access token][git_token] for your user (in your environment or your local `.env` file).
-  - This token is necessary for authenticating git requests made in a Docker/Podman container.
 - Membership in the `azure-sdk-write` GitHub group.
 
 Test recordings will be updated if tests are run while `AZURE_TEST_RUN_LIVE` is set to "true" and
@@ -409,15 +403,16 @@ The recording directory in this case is `2Km2Z8755`, the string between the two
 
 After verifying that your recording updates look correct, you can use the [`manage_recordings.py`][manage_recordings]
 script from `azure-sdk-for-python/scripts` to push these recordings to the `azure-sdk-assets` repo. This script accepts
-a verb and a **relative** path to your package's `assets.json` file. For example, from the root of the
-`azure-sdk-for-python` repo:
+a verb and a **relative** path to your package's `assets.json` file (this path is optional, and simply `assets.json`
+by default). For example, from the root of the `azure-sdk-for-python` repo:
 ```
-python scripts/manage_recordings.py push sdk/{service}/{package}/assets.json
+python scripts/manage_recordings.py push -p sdk/{service}/{package}/assets.json
 ```
 
 The verbs that can be provided to this script are "push", "restore", and "reset":
 - **push**: pushes recording updates to a new assets repo tag and updates the tag pointer in `assets.json`.
 - **restore**: fetches recordings from the assets repo, based on the tag pointer in `assets.json`.
+- **reset**: discards any pending changes to recordings, based on the tag pointer in `assets.json`.
 
 After pushing your recordings, the `assets.json` file for your package will be updated to point to a new `Tag` that
 contains the updates. Include this `assets.json` update in any pull request to update the recordings pointer in the
@@ -714,8 +709,6 @@ Tests that use the Shared Access Signature (SAS) to authenticate a client should
 [azure_portal]: https://portal.azure.com/
 [azure_recorded_test_case]: https://github.com/Azure/azure-sdk-for-python/blob/7e66e3877519a15c1d4304eb69abf0a2281773/tools/azure-sdk-tools/devtools_testutils/azure_recorded_testcase.py#L44
 
-[docker_install]: https://docs.docker.com/get-docker/
-
 [engsys_wiki]: https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/48/Create-a-new-Live-Test-pipeline?anchor=test-resources.json
 [env_var_loader]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/envvariable_loader.py
 
@@ -732,7 +725,6 @@ Tests that use the Shared Access Signature (SAS) to authenticate a client should
 [mgmt_settings_fake]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/mgmt_settings_fake.py
 
 [packaging]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/packaging.md
-[podman]: https://podman.io/
 [proxy_cert_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/documentation/test-proxy/trusting-cert-per-language.md
 [proxy_general_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md
 [proxy_migration_guide]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/test_proxy_migration_guide.md