fix: Improve E2E failure reporting in GitHub Actions

Added logic to capture and summarize failing tests from E2E runs.
This was implemented by redirecting script output to a log file and then
parsing this file for Go test failures.
The summary is included in the Slack notification when a job fails, providing
immediate insight into which specific tests failed for a given provider.

Fixes #2283

Co-authored-by: Cursor <cursor@users.noreply.github.com>
Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>

Author: chmouel

.github/workflows/e2e.yaml

@@ -175,7 +175,9 @@ jobs:
           TEST_BITBUCKET_SERVER_API_URL: ${{ secrets.BITBUCKET_SERVER_API_URL }}
           TEST_BITBUCKET_SERVER_WEBHOOK_SECRET: ${{ secrets.BITBUCKET_SERVER_WEBHOOK_SECRET }}
         run: |
-          ./hack/gh-workflow-ci.sh run_e2e_tests
+          set -o pipefail
+          LOG_FILE="/tmp/e2e-tests-${{ matrix.provider }}.log"
+          ./hack/gh-workflow-ci.sh run_e2e_tests 2>&1 | tee "${LOG_FILE}"
 
       - name: Run E2E Tests on nightly
         # This step still runs specifically for schedule or workflow_dispatch
@@ -194,7 +196,30 @@ jobs:
           TEST_BITBUCKET_SERVER_API_URL: ${{ secrets.BITBUCKET_SERVER_API_URL }}
           TEST_BITBUCKET_SERVER_WEBHOOK_SECRET: ${{ secrets.BITBUCKET_SERVER_WEBHOOK_SECRET }}
         run: |
-          ./hack/gh-workflow-ci.sh run_e2e_tests
+          set -o pipefail
+          LOG_FILE="/tmp/e2e-tests-${{ matrix.provider }}.log"
+          ./hack/gh-workflow-ci.sh run_e2e_tests 2>&1 | tee "${LOG_FILE}"
+
+      - name: Summarize failing tests
+        if: ${{ failure() }}
+        id: failing-tests
+        run: |
+          LOG_FILE="/tmp/e2e-tests-${{ matrix.provider }}.log"
+          if [[ ! -f "${LOG_FILE}" || ! -s "${LOG_FILE}" ]]; then
+            echo "slack_message=E2E job failed but no test log was captured for provider '${{ matrix.provider }}'." >> "${GITHUB_OUTPUT}"
+            exit 0
+          fi
+
+          mapfile -t FAILURES < <(grep -E '^--- FAIL: ' "${LOG_FILE}" | sed -E 's/^--- FAIL: ([^[:space:]]+).*/\1/' | sort -u)
+
+          if [[ ${#FAILURES[@]} -eq 0 ]]; then
+            echo "slack_message=E2E job failed without explicit go test failures; check workflow logs for provider '${{ matrix.provider }}'." >> "${GITHUB_OUTPUT}"
+            exit 0
+          fi
+
+          joined_failures=$(printf '%s, ' "${FAILURES[@]}")
+          joined_failures=${joined_failures%, }
+          echo "slack_message=Failing tests for provider ${{ matrix.provider }}: ${joined_failures}" >> "${GITHUB_OUTPUT}"
 
       - name: Collect logs
         if: ${{ always() }}
@@ -216,11 +241,14 @@ jobs:
           name: logs-e2e-tests-${{ matrix.provider }}
           path: /tmp/logs
 
-      - name: Report Status
+      - name: Report E2E test results to Slack
         if: ${{ always() && github.ref_name == 'main' && github.event_name == 'schedule' }}
         uses: ravsamhq/notify-slack-action@v2
         with:
           status: ${{ job.status }}
           notify_when: "failure"
+          message_format: |
+            {emoji} *{workflow}* {status_message} in <{repo_url}|{repo}@{branch}> on <{commit_url}|{commit_sha}>
+            ${{ steps.failing-tests.outputs.slack_message }}
         env:
           SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

docs/content/docs/guide/cli.md

@@ -458,7 +458,7 @@ tkn pac cel -b <body.json> -H <headers.txt>
 If run in a terminal, you'll get a prompt:
 
 ```console
-CEL expression> 
+CEL expression>
 ```
 
 * Use ↑/↓ arrows to navigate history.
@@ -485,7 +485,7 @@ echo 'event == "pull_request"' | tkn pac cel -b body.json -H headers.txt
 
 * **Webhook payload** (`body.*`): All fields from the webhook JSON.
 * **HTTP headers** (`headers.*`): All HTTP headers.
-* **Files** (`files.*`): Always empty in CLI mode.  
+* **Files** (`files.*`): Always empty in CLI mode.
   **Note:** `fileChanged`, `fileDeleted`, `fileModified` and similar functions are **not implemented yet** in the CLI.
 * **PAC Parameters** (`pac.*`): All variables for backward compatibility.
 

docs/content/docs/install/settings.md

@@ -158,7 +158,7 @@ There are a few things you can configure through the ConfigMap
   both a push event and a pull request. If a push event comes from a commit that is
   part of an open pull request, the push event will be skipped as it would create
   a duplicate PipelineRun.
-  
+
   This feature works by checking if a pushed commit SHA exists in any open pull request,
   and if so, skipping the push event processing.
 
@@ -207,12 +207,12 @@ Pipelines-as-Code supports fetching tasks and pipelines with its remote annotati
 * `hub-catalog-type`
 
   The type of hub catalog. Supported values are:
-  
+
   * `artifacthub` - For Artifact Hub (default if not specified)
   * `tektonhub` - For Tekton Hub
 
 * By default, both Artifact Hub and Tekton Hub are configured:
-  
+
   * Artifact Hub is the default catalog (no prefix needed, but `artifact://` can be used explicitly)
   * Tekton Hub is available using the `tektonhub://` prefix
 
@@ -223,7 +223,7 @@ Pipelines-as-Code supports fetching tasks and pipelines with its remote annotati
   catalog-1-name: "tekton"
   catalog-1-url: "https://api.custom.hub/v1"
   catalog-1-type: "tektonhub"
-  
+
   catalog-2-id: "artifact"
   catalog-2-name: "tekton-catalog-tasks"
   catalog-2-url: "https://artifacthub.io"