Idea: Monorepo for aws-durable-execution-sdk-python-testing and aws-durable-execution-sdk-python

[zhongkechen]

Currently there are two separate Github repos for AWS Durable Functions - aws-durable-execution-sdk-python (SDK repo for short) is for the runtime SDK and aws-durable-execution-sdk-python-testing (testing repo for short) for testing utilities.

Problems with Separate Repos

• Sync drift: SDK changes and their corresponding tests live in different repos, making it easy for tests to fall out of date or reference stale APIs (e.g., the imports from aws_durable_execution_sdk_python.state and aws_durable_execution_sdk_python.threading in your active test file could break silently if the SDK renames modules).
• Cross-repo CI complexity: Every SDK PR needs a way to trigger the test repo's CI against the proposed change.
• Atomic changes are impossible: A single feature (like the empty-checkpoint coalescing in issue #325) requires two separate PRs in two repos, reviewed and merged in the right order. If the test PR merges first, CI breaks; if the SDK PR merges first, there's a window with no test coverage.
• Dependency management overhead: The test repo must declare a dependency on a specific SDK version/branch. This means maintaining version pins, dealing with pre-release builds, or pointing at git refs — all of which add friction.
• Discoverability: New contributors have to find and clone two repos. The relationship between a feature and its tests isn't obvious from either repo alone.
• Inconsistent tooling: Linting, formatting, and CI configs diverge over time since each repo has its own setup.

Recommended solution: monorepo

Hatch supports monorepo:

• https://hatch.pypa.io/1.16/how-to/environment/workspace/

[yaythomas]

Just thinking through how will versioning & publishing separately to aws-durable-execution-sdk-python and aws-durable-execution-sdk-python-testing work?

  monorepo-root/
  ├── aws-durable-execution-sdk-python/
  │   ├── pyproject.toml
  │   └── src/
  │       └── aws_durable_execution_sdk/
  │           ├── __init__.py
  │           └── __about__.py
  ├── aws-durable-execution-sdk-python-testing/
  │   ├── pyproject.toml
  │   └── src/
  │       └── aws_durable_execution_sdk_testing/
  │           ├── __init__.py
  │           └── __about__.py

Tagging a release would have to take a specific format, like

aws-durable-execution-sdk-python/v1.2.3
aws-durable-execution-sdk-python-testing/v3.1.2

Even so, the tag would then apply to the entire repo, i.e you're effectively tagging SDK code when you tag a testing release.

Then there would have to be some parsing in the gh action [published]

1. check tag and parse out package name
2. run hatch build in that subdir
3. selectively publish specific sub dir per https://github.com/pypa/gh-action-pypi-publish#customizing-target-package-dists-directory

[zhongkechen]

Just use one version number for all packages in monorepo.

[bchampp]

It's not as simple as this. We need to maintain independent versions between the language SDK and testing library. We also have much more freedom and autonomy for how we treat version bumps in the testing library as opposed to the SDK since the SDK is bundled in the lambda runtime.

[bchampp]

I'm in favour of a monorepo, but there are some design constraints we faced when setting the project structure up that we need to remain cognizant of. If hatch has the tooling to support this while allowing isolated versioning and publishing across the modules, then great, I'm all for it.

I agree this will simplify how we test code, and allow us to atomically release changes together (which we can already do in the JS SDK).