Skip to content

Commit f7bd4ad

Browse files
authored
Merge pull request #1 from grove-platform/critical-enh
Critical Enhancements & Bug Fixes
2 parents 766dd6a + 03d5231 commit f7bd4ad

62 files changed

Lines changed: 2265 additions & 621 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/ci.yml

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
jobs:
10+
test:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v4
14+
15+
- uses: actions/setup-go@v5
16+
with:
17+
go-version: '1.24'
18+
19+
- name: Download dependencies
20+
run: go mod download
21+
22+
- name: Run tests
23+
# Note: -race disabled due to pre-existing race conditions in tests that spawn
24+
# background goroutines. These should be fixed by adding proper synchronization.
25+
run: go test -v ./...
26+
27+
lint:
28+
runs-on: ubuntu-latest
29+
steps:
30+
- uses: actions/checkout@v4
31+
32+
- uses: actions/setup-go@v5
33+
with:
34+
go-version: '1.24'
35+
36+
- name: golangci-lint
37+
uses: golangci/golangci-lint-action@v6
38+
with:
39+
version: latest
40+
41+
security:
42+
runs-on: ubuntu-latest
43+
steps:
44+
- uses: actions/checkout@v4
45+
46+
- uses: actions/setup-go@v5
47+
with:
48+
go-version: '1.24'
49+
50+
- name: Run gosec
51+
uses: securego/gosec@master
52+
with:
53+
# Exclude G101 (hardcoded credentials - false positive on env var names)
54+
# Exclude G115 (integer overflow - false positive for PR numbers)
55+
# Exclude G304 (file inclusion - intentional for CLI tools)
56+
# Exclude G306 (file permissions - config files don't need 0600)
57+
args: -exclude=G101,G115,G304,G306 ./...
58+
59+
build:
60+
runs-on: ubuntu-latest
61+
needs: [test, lint]
62+
steps:
63+
- uses: actions/checkout@v4
64+
65+
- uses: actions/setup-go@v5
66+
with:
67+
go-version: '1.24'
68+
69+
- name: Build
70+
run: go build -v ./...
71+
72+
deploy:
73+
runs-on: ubuntu-latest
74+
needs: [build, security]
75+
# Only deploy on push to main (not on PRs)
76+
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
77+
78+
permissions:
79+
contents: read
80+
id-token: write # Required for Workload Identity Federation
81+
82+
env:
83+
PROJECT_ID: "github-copy-code-examples"
84+
SERVICE_NAME: "examples-copier"
85+
REGION: "us-central1"
86+
87+
steps:
88+
- uses: actions/checkout@v4
89+
90+
- name: Authenticate to Google Cloud
91+
uses: google-github-actions/auth@v2
92+
with:
93+
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
94+
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
95+
96+
- name: Set up Cloud SDK
97+
uses: google-github-actions/setup-gcloud@v2
98+
99+
- name: Deploy to Cloud Run
100+
run: |
101+
gcloud run deploy $SERVICE_NAME \
102+
--source . \
103+
--region $REGION \
104+
--project $PROJECT_ID \
105+
--allow-unauthenticated \
106+
--env-vars-file=env-cloudrun.yaml \
107+
--max-instances=10 \
108+
--cpu=1 \
109+
--memory=512Mi \
110+
--timeout=300s \
111+
--concurrency=80 \
112+
--port=8080 \
113+
--platform=managed
114+
115+
- name: Show deployment URL
116+
run: |
117+
URL=$(gcloud run services describe $SERVICE_NAME \
118+
--region $REGION \
119+
--project $PROJECT_ID \
120+
--format='value(status.url)')
121+
echo "🚀 Deployed to: $URL"
122+

.gitignore

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Binaries
2-
examples-copier
2+
github-copier
33
code-copier
44
copier
55
*.exe
@@ -23,19 +23,21 @@ go.work
2323
# Environment files with secrets (working files - create from templates in configs/)
2424
# Working files (create from templates):
2525
# env.yaml - App Engine deployment config (from configs/env.yaml.*)
26-
# env-cloudrun.yaml - Cloud Run deployment config (from configs/env.yaml.*)
2726
# .env - Local development config (from configs/.env.local.example)
27+
# .env.test - Test config (from testdata/.env.test)
28+
# Note: env-cloudrun.yaml is committed (contains no secrets, only Secret Manager references)
2829
env.yaml
29-
env-cloudrun.yaml
3030
.env
3131
.env.local
32+
.env.test
3233
.env.production
3334
.env.*.local
3435

35-
# Explicitly keep template files in configs/ directory (these should be tracked)
36+
# Explicitly keep template files (these should be tracked)
3637
!configs/env.yaml.example
3738
!configs/env.yaml.production
3839
!configs/.env.local.example
40+
!testdata/.env.test
3941

4042
# Private keys
4143
*.pem
@@ -58,3 +60,4 @@ Thumbs.db
5860
# Temporary files
5961
tmp/
6062
temp/
63+
RECOMMENDATIONS.md

.pre-commit-config.yaml

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
repos:
2+
# Secrets detection
3+
- repo: https://github.com/gitleaks/gitleaks
4+
rev: v8.21.2
5+
hooks:
6+
- id: gitleaks
7+
8+
# Go linting
9+
- repo: https://github.com/golangci/golangci-lint
10+
rev: v1.62.2
11+
hooks:
12+
- id: golangci-lint
13+
14+
# Local Go hooks
15+
- repo: local
16+
hooks:
17+
- id: go-fmt
18+
name: go fmt
19+
entry: gofmt -w
20+
language: system
21+
types: [go]
22+
23+
- id: go-vet
24+
name: go vet
25+
entry: go vet ./...
26+
language: system
27+
pass_filenames: false
28+
types: [go]
29+
30+
- id: go-build
31+
name: go build
32+
entry: go build ./...
33+
language: system
34+
pass_filenames: false
35+
types: [go]
36+

AGENT.md

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Agent Context: GitHub Copier
2+
3+
Webhook service: PR merged → match files → transform paths → copy to target repos.
4+
5+
## File Map
6+
7+
```
8+
app.go # entrypoint, HTTP server
9+
services/
10+
webhook_handler_new.go # HandleWebhookWithContainer()
11+
workflow_processor.go # ProcessWorkflow() - core logic
12+
pattern_matcher.go # MatchFile(pattern, path) bool
13+
github_auth.go # ConfigurePermissions() error
14+
github_read.go # GetFilesChangedInPr(), RetrieveFileContents()
15+
github_write_to_target.go # AddFilesToTargetRepoBranch()
16+
github_write_to_source.go # UpdateDeprecationFile()
17+
file_state_service.go # tracks upload/deprecate queues
18+
main_config_loader.go # LoadConfig() with $ref support
19+
service_container.go # DI container
20+
types/
21+
config.go # Workflow, Transformation, SourcePattern structs
22+
types.go # ChangedFile, UploadKey, UploadFileContent
23+
configs/environment.go # Config struct, LoadEnvironment()
24+
tests/utils.go # test helpers, httpmock setup
25+
```
26+
27+
## Key Types
28+
29+
```go
30+
// types/config.go
31+
type PatternType string // "prefix" | "glob" | "regex"
32+
type TransformationType string // "move" | "copy" | "glob" | "regex"
33+
34+
type Workflow struct {
35+
Name string
36+
Source SourceConfig // Repo, Branch, Patterns []SourcePattern
37+
Destination DestinationConfig // Repo, Branch
38+
Transformations []Transformation // Type, From, To, Pattern, Replacement
39+
Commit CommitConfig // Strategy, Message, PRTitle, AutoMerge
40+
}
41+
42+
// types/types.go
43+
type ChangedFile struct { Path, Status string } // Status: "ADDED"|"MODIFIED"|"DELETED"
44+
type UploadKey struct { RepoName, BranchPath string }
45+
```
46+
47+
## Global State (⚠️ mutable)
48+
49+
```go
50+
// services/github_write_to_target.go
51+
var FilesToUpload map[UploadKey]UploadFileContent
52+
// services/github_auth.go
53+
var InstallationAccessToken string
54+
var OrgTokens map[string]string
55+
```
56+
57+
## Config Example
58+
59+
```yaml
60+
workflows:
61+
- name: "sync-docs"
62+
source: { repo: "org/src", branch: "main", patterns: [{type: glob, pattern: "docs/**"}] }
63+
destination: { repo: "org/dest", branch: "main" }
64+
transformations: [{ type: move, from: "docs/", to: "public/" }]
65+
commit: { strategy: pr, message: "Sync" } # strategy: direct|pr
66+
```
67+
68+
## Test Commands
69+
70+
```bash
71+
go test ./... # all
72+
go test ./services/... -run TestWorkflow -v # specific
73+
```
74+
75+
## Edit Patterns
76+
77+
| Task | Files to modify |
78+
|------|-----------------|
79+
| New transformation | `types/config.go` (TransformationType) → `workflow_processor.go` (processFileForWorkflow) |
80+
| New pattern type | `types/config.go` (PatternType) → `pattern_matcher.go` |
81+
| New config field | `types/config.go` (struct) → consumers in `workflow_processor.go` |
82+
| Webhook logic | `webhook_handler_new.go` |
83+
84+
## Conventions
85+
86+
- Return `error`, never `log.Fatal`
87+
- Wrap errors: `fmt.Errorf("context: %w", err)`
88+
- Nil-check GitHub API responses before dereference
89+
- Tests use `httpmock`; see `tests/utils.go`
90+
- **Changelog**: Update `CHANGELOG.md` for all notable changes (follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/))

CHANGELOG.md

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Changelog
2+
3+
All notable changes to this project will be documented in this file.
4+
5+
## 17 Dec 2025
6+
7+
### Added
8+
- CI/CD pipeline with GitHub Actions (`.github/workflows/ci.yml`)
9+
- Test job
10+
- Lint job with golangci-lint
11+
- Security scanning with gosec
12+
- Build verification
13+
- Automated deployment to Cloud Run on merge to main (via Workload Identity Federation)
14+
- Pre-commit hooks for secrets detection and Go linting (`.pre-commit-config.yaml`)
15+
- AGENT.md for AI agent context
16+
- Comprehensive test suite for `workflow_processor.go` (843 lines, 94%+ coverage)
17+
- Integration test harness for local testing (`scripts/integration-test.sh`)
18+
- Test environment configuration (`testdata/.env.test`)
19+
20+
### Changed
21+
- Renamed module from `github.com/mongodb/code-example-tooling/code-copier` to `github.com/grove-platform/github-copier`
22+
- Renamed binary from `examples-copier` to `github-copier`
23+
- Renamed `test-payloads/` to `testdata/` (Go convention)
24+
- All `log.Fatal` calls replaced with proper error returns for graceful error handling
25+
- `FileStateService.filesToDeprecate` changed from single-entry map to slice-based accumulation
26+
27+
### Fixed
28+
- Deprecation file accumulation bug: multiple deprecated files now correctly accumulate instead of overwriting
29+
- Nil pointer dereference bugs across GitHub API calls in:
30+
- `services/github_read.go`
31+
- `services/github_write_to_source.go`
32+
- `services/main_config_loader.go`
33+
- `services/config_loader.go`
34+
- DELETED file status handling: GitHub GraphQL API returns uppercase `DELETED` but code checked for lowercase `removed`
35+
- Graceful shutdown now properly waits for in-flight requests and cleans up resources
36+
37+
### Security
38+
- Added gitleaks pre-commit hook for secrets detection
39+
- Added gosec security scanning in CI pipeline
40+
41+
## Initial Release (Migration from mongodb/code-example-tooling)
42+
43+
### Features
44+
- Webhook service for automated file copying on PR merge
45+
- Pattern matching support: prefix, glob, regex
46+
- Transformation types: move, copy, glob, regex
47+
- Main config system with `$ref` support for distributed workflow configs
48+
- Commit strategies: direct commit or pull request
49+
- Health and metrics endpoints
50+
- Slack notifications for operational visibility
51+
- MongoDB audit logging (optional)
52+
- Google Cloud Logging integration
53+
- Dry-run mode for testing
54+

0 commit comments

Comments
 (0)