Skip to content

README.md

Latest commit

 

History

History
64 lines (55 loc) · 4.26 KB

README.md

File metadata and controls

64 lines (55 loc) · 4.26 KB

Info about e2e tests:

  • We run the re-usable workflow and we verify using the slsa-verifier at HEAD and for all previous releases. This ensures that any upcoming re-usable workflow (builder) release is compatible (i.e., can be verified) using any prior version of the slsa-verifier. See e2e-verify.common.sh

  • Each file is named e2e.<ecosystem>.<trigger>.<branch>.<config-parameters>.slsa<level>.yml. There is one exception to this: the tag trigger: there's not tag trigger per se, but we use it to differentiate it from a push.

  • Each workflow runs on a schedule event. For push, we create a commit (see e2e-push.sh); for tags we create a version (see e2e-create-release.sh), for dispatch we create a workflow_dispatch (see e2e-dispatch.sh) etc. Each test that creates a version must define a unique DEFAULT_VERSION environment variable which sets the major version used by that test.

  • Each push has a commit message containing the name of the workflow it's intended for ($THIS_FILE in workflow definitions), to ensure other workflows triggered by a commit don't run - this is especially useful when we update the repo's code manually for fixes. To ensure each new commit updates the code, we use the date/time the workflow is run and update a file within the e2e/ folder: the file is the name of the workflow appended with the .txt, e.g., e2e/e2e.go.push.main.config-noldflags.slsa3.yml.txt, and it contains the date/time the commit is created.

  • Each version created has the workflow name ($THIS_FILE) as part of the release notes. This allows workflow to exit early if the release is not intended for them, e.g. different branch, etc. The version created is the same as the latest version +1 for the PATCH. To ensure workflows don't race on release version, they each use a different MAJOR. (v10 - v23 as of today). The first run will use the version as defined by DEFAULT_VERSION in the workflows.

  • When a workflow does not complete with the expected outcome (fail instead of success; or success instead of failure), we create an issue on the slsa-framework-slsa-github-generator-go repo. Upon new failures, we append a comment to an existing opened issue (see e2e-report-failure.sh). When the tests are fixed and succeed again, we close the open issues automatically (see e2e-report-success.sh). We identify existing opened issues via their body which contains the workflow file $THIS_FILE. When creating an issue, we label them (see e2e-report-failure.sh): it's important the labels exist on the repository the issues are reported; or it will fail.

  • We have adversarial tests (<config-parameters> = adversarial-XXX) that should fail. They use a rough time-estimate to start tampering with the artifacts used by the builder. This can be a bit flaky at times.

  • When you update the main branch, try to regularly update the branch1 manually. You can use bash sync-branch1.sh for that.

  • We have scripts to run the e2e tests via CLI: manual-run-all-e2e.sh to run them all, manual-run-e2e.sh to run a single one of them.
⚡