This document is the Primary Heuristic for AI agents. It defines the systemic boundaries, operational standards, and institutional memory required to contribute effectively to the Observability Hub.
Agents must distinguish between the two primary orchestration tiers to avoid "circular dependencies" and resource contention.
- Host Tier (Systemd): Reserved for hardware-level telemetry, security gates, and GitOps reconciliation. Reliability here is critical for cluster recovery. Core logic is strictly encapsulated in
internal/to ensure reusability and enforce project boundaries. - Cluster Tier (K3s): Handles scalable data services (Postgres, Loki, Prometheus, Grafana, Tempo, MinIO). Orchestrated via OpenTofu (IaC) in
tofu/.
cmd/: Minimal entry points for services. Focuses on configuration and orchestration.cmd/web/: Static site generator entry point.cmd/proxy/: API Gateway and GitOps webhook listener entry point.cmd/worker/: Unified one-shot execution engine for Analytics and Ingestion tasks.
internal/: Private Implementation Layer. Enforces Go's internal package visibility rules.k3s/: Kubernetes manifests and Helm values for the data platform.makefiles/: Modular logic for the root automation layer.systemd/: Host-tier unit files for production service management.scripts/: Operational utilities (Traffic gen, ADR creation, Tailscale gate).docs/: Institutional memory. ADRs, Architecture, Incidents, and Notes.
The project uses a unified automation layer. Always prefer make commands as they handle environment wrapping (Nix) and consistency checks automatically.
| Domain | Command | Description |
|---|---|---|
| Governance | make adr |
Creates a new Architecture Decision Record. |
| IaC | tofu plan / apply |
Manages K3s data services and infrastructure state. |
| Quality | make lint |
Lints markdown and configuration files. |
| Go Dev | make test |
Runs full test suite across the monorepo. |
| Security | make vuln-scan |
Executes govulncheck for dependency auditing. |
| K3s Ops | make build-postgres |
Builds and imports the custom PG image into K3s. |
| Host Ops | make proxy-build |
Builds proxy server to bin/ and restarts the service. |
- Thin Main: Entry points in
cmd/must be minimal. Move all core domain logic tointernal/. - Internal-First: Shared libraries reside in
internal/to prevent external logic leakage. - Environment Loading: Always use
internal/envfor standardized.envdiscovery. - Observability: Every service must emit structured JSON logs using
internal/telemetry. - Telemetry: All instrumentation must be handled through the centralized
internal/telemetrylibrary. - Testing: Table-driven tests are the standard. Run
make go-covto verify coverage.
- ADRs (
docs/decisions/): Mandatory for any architectural pivot. - RCA (
docs/incidents/): Document every failure to prevent regression. - Golden Path: Maintain
docs/workflows.mdto reflect the CI/CD reality.
- Secrets: NEVER commit secrets. Use
.envfor local dev and OpenBao for production. - GitOps: Host-tier changes are applied via
gitops_sync.sh(triggered by Proxy webhooks). - Security: All Kubernetes manifests must pass
kube-lint. All Go code must passgo-vuln-scan.