Draft
Conversation
Signed-off-by: Greg Schohn <schohn@amazon.com>
Signed-off-by: Greg Schohn <schohn@amazon.com>
…ation-tests; Phases 0-5) See docs/e2eOrchestrationImplementationPlan.md. Implements the planning/expansion side of the orchestration test framework described in docs/e2eOrchestrationTestDesign.md. Given a matrix spec (focus component + change classes + dependency patterns), the framework expands it into concrete test cases with topology-driven expectations. Components: - checksumReporter: runs the config processor and extracts per-component config checksums, downstream checksum projections, and dependency edges - derivedSubgraph: classifies components relative to a focus into immediate dependents, transitive dependents, upstream prerequisites, and independent branches - approvedMutators: curated registry of reviewed config mutations - matrixExpander: expands matrix specs into concrete test cases, deriving reran/unchanged/blockedOn expectations from dependency topology and validating that tagged mutators actually affect the expected target. Enforces requireFullCoverage. - reportSchema: structured pass/fail results with human-readable output Includes a demo script (`npx tsx src/demo.ts`) showing the pipeline from base config through expansion to simulated test execution. Does not yet include the runtime side (Phases 6-10): ConfigMap state handoff, assert container, outer Argo workflow, or teardown. Signed-off-by: Greg Schohn <schohn@amazon.com>
Extend @opensearch-migrations/e2e-orchestration-tests with the first runtime-side scaffolding for matrix-driven orchestration tests. Adds: - assertLogic: pure assertion logic for baseline/noop/mutated runs, checking expected checksums and terminal phases - stateStore: ConfigMap naming/helpers for checksum and observation handoff between runs - buildOuterWorkflow: outer Argo workflow generator for a 3-run flow (baseline, noop, mutate) plus teardown - demo updates to show workflow generation and simulated assert behavior Also tightens the contract between expansion and runtime: - ExpandedTestCase now carries mutatedConfig - runtime expectations are modeled explicitly as allCompleted, allSkipped, or selective rerun/unchanged expectations This is still scaffold, not a fully runnable end-to-end implementation: - inner submission is still a placeholder for `workflow configure` - the `e2e-assert` runtime binary is not built yet - CRD teardown is still TODO Signed-off-by: Greg Schohn <schohn@amazon.com>
Extend @opensearch-migrations/e2e-orchestration-tests with a live integration runner that executes a generated matrix test case against a real cluster through the migration console workflow CLI. Adds: - e2e-run.ts to drive baseline/noop/mutated runs using `workflow configure edit --stdin` and `workflow submit` - live workflow waiting with stall detection and failed-node reporting - CRD observation collection for proxy, snapshot, snapshot migration, and replay resources - runtime assertions over observed phases and config checksums - teardown that cleans up the submitted workflow and all CRD resources discovered across runs Also improves runner safety by: - matching the known working `workflow submit` invocation - running cleanup in `finally` - documenting the shared `migration-workflow` name constraint This is still a manual/live integration path, not a generalized outer workflow implementation. Signed-off-by: Greg Schohn <schohn@amazon.com>
gregschohn
temporarily deployed
to
migrations-cicd
GitHub Actions
Inactive
gregschohn
temporarily deployed
to
migrations-cicd
GitHub Actions
Inactive
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| for (let i = 0; i < parts.length - 1; i++) { | ||
| current = (current as Record<string | number, unknown>)[parts[i]]; | ||
| } | ||
| (current as Record<string | number, unknown>)[parts[parts.length - 1]] = value; |
| for (let i = 0; i < parts.length - 1; i++) { | ||
| current = (current as Record<string | number, unknown>)[parts[i]]; | ||
| } | ||
| (current as Record<string | number, unknown>)[parts[parts.length - 1]] = value; |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2690 +/- ##
============================================
- Coverage 72.52% 71.90% -0.62%
Complexity 90 90
============================================
Files 709 638 -71
Lines 32988 25612 -7376
Branches 2823 2839 +16
============================================
- Hits 23925 18417 -5508
+ Misses 7784 5917 -1867
+ Partials 1279 1278 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
… goes wrong, dump everything k8s, like what we do for python E2E tests. Signed-off-by: Greg Schohn <schohn@amazon.com>
gregschohn
temporarily deployed
to
migrations-cicd
GitHub Actions
Inactive
gregschohn
temporarily deployed
to
migrations-cicd
GitHub Actions
Inactive
Extend @opensearch-migrations/e2e-orchestration-tests with a spec-driven runtime path for live and generated workflow execution. Adds: - JSON test spec loading for base config, matrix selection, cleanup, and approval-gate definitions - fixture registry support so specs can reference named setup, cleanup, and validation actions - approval-gate fixtures with index and doc-count validation hooks - a sample proxy-focused scenario for fullMigrationWithTraffic - a generate-outer entrypoint for rendering outer workflow YAML from a spec Updates: - wire the live runner to load specs, honor approval gates, and support optional pause-before-cleanup inspection - expand the generated outer workflow to include pre-run cleanup, baseline/noop/mutate runs, gate monitoring, assertions, suspend-before-teardown, and full teardown - update the sample fullMigrationWithTraffic config for in-cluster localstack addressing - refresh checksumReporter snapshots for the updated sample config Signed-off-by: Greg Schohn <schohn@amazon.com>
These are much clearer and up-to-date with where the branch is at this point of time. Signed-off-by: Greg Schohn <schohn@amazon.com>
gregschohn
had a problem deploying
to
migrations-cicd
GitHub Actions
Failure
gregschohn
had a problem deploying
to
migrations-cicd
GitHub Actions
Failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This builds upon #2447 to provide some E2E testing for workflows. It is still a WIP.
Here's the embedded design for this testing framework.
Issues Resolved
Testing
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.