Skip to content

Commit 17877a0

Browse files
isaiah-grafanaisaiah-grafana
committed
refactor(trufflehog): prefixes.txt + builder for exclude rules
- Add trufflehog/prefixes.txt (plain directory prefixes) and build_exclude_file.py - Regenerate global-exclude.txt; workflow can rebuild from script+prefixes if artifact missing - Last-resort heredoc synced to builder output; STATIC_PATTERNS live in the script Made-with: Cursor
1 parent f89f7b4 commit 17877a0

4 files changed

Lines changed: 118 additions & 25 deletions

File tree

‎.github/workflows/reusable-trufflehog.yml‎

Lines changed: 28 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -57,22 +57,43 @@ jobs:
5757
DEST=/tmp/trufflehog-exclude.txt
5858
REPO=grafana/security-github-actions
5959
REF=main
60-
# Load trufflehog/global-exclude.txt from main (edit that file for org-wide excludes).
61-
RAW_URL="https://raw.githubusercontent.com/grafana/security-github-actions/${REF}/trufflehog/global-exclude.txt"
60+
# Prefer trufflehog/global-exclude.txt on main. To change rules: edit trufflehog/prefixes.txt
61+
# and/or trufflehog/build_exclude_file.py, then: python3 trufflehog/build_exclude_file.py > trufflehog/global-exclude.txt
62+
RAW_BASE="https://raw.githubusercontent.com/grafana/security-github-actions/${REF}/trufflehog"
63+
RAW_URL="${RAW_BASE}/global-exclude.txt"
64+
RAW_BUILD="${RAW_BASE}/build_exclude_file.py"
65+
RAW_PREFIXES="${RAW_BASE}/prefixes.txt"
6266
if gh api "repos/${REPO}/contents/trufflehog/global-exclude.txt?ref=${REF}" \
6367
-H "Accept: application/vnd.github.v3.raw" -o "${DEST}" 2>/dev/null && [[ -s "${DEST}" ]]; then
6468
echo "Loaded exclude patterns from ${REPO}@${REF} (GitHub API)"
6569
elif curl -fsSL "${RAW_URL}" -o "${DEST}" 2>/dev/null && [[ -s "${DEST}" ]]; then
6670
echo "Loaded exclude patterns from raw.githubusercontent.com (${REF})"
71+
elif gh api "repos/${REPO}/contents/trufflehog/build_exclude_file.py?ref=${REF}" \
72+
-H "Accept: application/vnd.github.v3.raw" -o /tmp/trufflehog_build_exclude_file.py 2>/dev/null \
73+
&& gh api "repos/${REPO}/contents/trufflehog/prefixes.txt?ref=${REF}" \
74+
-H "Accept: application/vnd.github.v3.raw" -o /tmp/trufflehog_prefixes.txt 2>/dev/null \
75+
&& python3 /tmp/trufflehog_build_exclude_file.py /tmp/trufflehog_prefixes.txt > "${DEST}" 2>/dev/null \
76+
&& [[ -s "${DEST}" ]]; then
77+
echo "Built exclude file from build_exclude_file.py + prefixes.txt (${REPO}@${REF}, GitHub API)"
78+
elif curl -fsSL "${RAW_BUILD}" -o /tmp/trufflehog_build_exclude_file.py 2>/dev/null \
79+
&& curl -fsSL "${RAW_PREFIXES}" -o /tmp/trufflehog_prefixes.txt 2>/dev/null \
80+
&& python3 /tmp/trufflehog_build_exclude_file.py /tmp/trufflehog_prefixes.txt > "${DEST}" 2>/dev/null \
81+
&& [[ -s "${DEST}" ]]; then
82+
echo "Built exclude file from build_exclude_file.py + prefixes.txt (raw GitHub)"
6783
else
68-
echo "::warning::Could not fetch trufflehog/global-exclude.txt from ${REPO}@${REF} (not on branch yet, or GITHUB_TOKEN cannot read that repo). Using bundled fallback — merge exclusions to main or allow workflows to read internal repos."
69-
# Keep in sync with trufflehog/global-exclude.txt (used only when fetch fails).
84+
echo "::warning::Could not fetch or rebuild TruffleHog excludes from ${REPO}@${REF}. Using last-resort bundled file — merge to main or fix token access."
85+
# Last resort only: must match stdout of python3 trufflehog/build_exclude_file.py (same commit).
7086
cat > "${DEST}" <<'EOF'
71-
# TruffleHog --exclude-paths: one Go regexp per non-blank, non-# line.
87+
# Generated by trufflehog/build_exclude_file.py + trufflehog/prefixes.txt — do not edit by hand.
88+
# Regenerate: python3 trufflehog/build_exclude_file.py > trufflehog/global-exclude.txt
7289
#
73-
# To change org-wide behavior: edit this file and merge to security-github-actions main.
74-
# Consumer repos load it at runtime via the GitHub API (no changes needed in those repos).
90+
# --- directory prefixes (from prefixes.txt) ---
91+
# prefix: vendor
92+
(^|\./|[/\\])vendor([/\\]|$)
93+
# prefix: content/grafana/dashboards
94+
(^|\./|[/\\])content/grafana/dashboards([/\\]|$)
7595
96+
# --- static path patterns ---
7697
# Lock files and checksums (contain hashes, not secrets)
7798
path:go\.sum$
7899
path:go\.mod$
@@ -90,13 +111,6 @@ jobs:
90111
91112
# Grafana plugin metadata
92113
path:grafana\.json$
93-
94-
# Go vendor/ directory (standard Go vendoring): third-party code and large data
95-
# files (e.g. golang.org/x/net/publicsuffix) are not repo secrets; TruffleHog often false-positives there.
96-
(^|\./|[/\\])vendor([/\\]|$)
97-
98-
# User-supplied Grafana dashboards checked into site repos (e.g. *.md / JSON); not org-managed secrets.
99-
(^|\./|[/\\])content/grafana/dashboards([/\\]|$)
100114
EOF
101115
fi
102116
echo "--- effective exclude file ---"

‎trufflehog/build_exclude_file.py‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
#!/usr/bin/env python3
2+
"""
3+
Emit TruffleHog --exclude-paths file content to stdout.
4+
5+
Edit trufflehog/prefixes.txt, then:
6+
python3 trufflehog/build_exclude_file.py > trufflehog/global-exclude.txt
7+
8+
If you change STATIC_PATTERNS or prefix rules, refresh the last-resort heredoc in
9+
.github/workflows/reusable-trufflehog.yml to match this script's stdout (or rely on
10+
fetch / rebuild-from-prefixes paths in CI).
11+
"""
12+
13+
from __future__ import annotations
14+
15+
import re
16+
import sys
17+
from pathlib import Path
18+
19+
# Fixed patterns (Go regexp syntax, one logical rule per line after strip).
20+
STATIC_PATTERNS = r"""
21+
# Lock files and checksums (contain hashes, not secrets)
22+
path:go\.sum$
23+
path:go\.mod$
24+
25+
# Dependency manifests (contain URLs that trigger false positives)
26+
path:package\.json$
27+
path:package-lock\.json$
28+
path:pnpm-lock\.yaml$
29+
path:yarn\.lock$
30+
path:poetry\.lock$
31+
path:Pipfile\.lock$
32+
path:uv\.lock$
33+
path:Cargo\.lock$
34+
path:Gemfile\.lock$
35+
36+
# Grafana plugin metadata
37+
path:grafana\.json$
38+
"""
39+
40+
41+
def prefix_to_regex(prefix: str) -> str:
42+
"""Match a path segment path in repo-relative, ./, and absolute CI paths."""
43+
p = prefix.strip().strip("/").replace("\\", "/")
44+
if not p:
45+
return ""
46+
escaped = re.escape(p)
47+
return rf"(^|\./|[/\\]){escaped}([/\\]|$)"
48+
49+
50+
def main() -> None:
51+
here = Path(__file__).resolve().parent
52+
prefixes_path = Path(sys.argv[1]) if len(sys.argv) > 1 else here / "prefixes.txt"
53+
54+
lines: list[str] = []
55+
lines.append(
56+
"# Generated by trufflehog/build_exclude_file.py + trufflehog/prefixes.txt — do not edit by hand.\n"
57+
"# Regenerate: python3 trufflehog/build_exclude_file.py > trufflehog/global-exclude.txt\n"
58+
"#\n"
59+
)
60+
61+
lines.append("# --- directory prefixes (from prefixes.txt) ---\n")
62+
for raw in prefixes_path.read_text().splitlines():
63+
stripped = raw.strip()
64+
if not stripped or stripped.startswith("#"):
65+
continue
66+
rx = prefix_to_regex(stripped)
67+
if rx:
68+
lines.append(f"# prefix: {stripped}\n{rx}\n")
69+
70+
lines.append("\n# --- static path patterns ---\n")
71+
lines.append(STATIC_PATTERNS.strip() + "\n")
72+
73+
sys.stdout.write("".join(lines))
74+
75+
76+
if __name__ == "__main__":
77+
main()

‎trufflehog/global-exclude.txt‎

Lines changed: 8 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,13 @@
1-
# TruffleHog --exclude-paths: one Go regexp per non-blank, non-# line.
1+
# Generated by trufflehog/build_exclude_file.py + trufflehog/prefixes.txt — do not edit by hand.
2+
# Regenerate: python3 trufflehog/build_exclude_file.py > trufflehog/global-exclude.txt
23
#
3-
# To change org-wide behavior: edit this file and merge to security-github-actions main.
4-
# Consumer repos load it at runtime via the GitHub API (no changes needed in those repos).
4+
# --- directory prefixes (from prefixes.txt) ---
5+
# prefix: vendor
6+
(^|\./|[/\\])vendor([/\\]|$)
7+
# prefix: content/grafana/dashboards
8+
(^|\./|[/\\])content/grafana/dashboards([/\\]|$)
59

10+
# --- static path patterns ---
611
# Lock files and checksums (contain hashes, not secrets)
712
path:go\.sum$
813
path:go\.mod$
@@ -20,11 +25,3 @@ path:Gemfile\.lock$
2025

2126
# Grafana plugin metadata
2227
path:grafana\.json$
23-
24-
# Go vendor/ directory (standard Go vendoring): third-party code and large data
25-
# files (e.g. golang.org/x/net/publicsuffix) are not repo secrets; TruffleHog often false-positives there.
26-
# Allow repo-relative paths that start with "./" (common on Actions checkout).
27-
(^|\./|[/\\])vendor([/\\]|$)
28-
29-
# User-supplied Grafana dashboards checked into site repos (e.g. *.md / JSON); not org-managed secrets.
30-
(^|\./|[/\\])content/grafana/dashboards([/\\]|$)

‎trufflehog/prefixes.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# One repo-relative directory prefix per line (forward slashes). No regex.
2+
# Regenerate: python3 trufflehog/build_exclude_file.py > trufflehog/global-exclude.txt
3+
4+
vendor
5+
content/grafana/dashboards

0 commit comments

Comments
 (0)
⚡