feat: improve observability

[jeluard]

General cleanup/improvements of the observability stack. For more context about those changes please refer to this document.

• introduced a dedicated crate amaru-observability
• introduced a schemas concept alongside trace and augment-trace macros (see schemas.rs)
• docs/TRACES.md documentation is automatically generated and checked by CI
• docs/traces-schema.json JSON Schema is automatically generated and checked by CI

Defining AMARU_TRACE_NOOP allows to disable all trace generations, and removes all associated code generation.

Syntax

(from the EDR)

#[trace(ledger::state::APPLY_BLOCK)]
fn apply_block(block: &Block) {
    // Record additional fields with schema context
    trace_record!(ledger::state::APPLY_BLOCK, block_size = block.size(), tx_count = block.transactions.len());

    // or alternatively
    trace_span!(ledger::state::APPLY_BLOCK);
}

This PR leverages significantly complex macros to generate meaningful compile-time errors. It received some AI love in the process.

Fixes #640

Summary by CodeRabbit

• New Features

  • Introduces a schema-driven tracing system with macros, a runtime registry, examples, and a CLI command to dump/export trace schemas.

• Documentation

  • Adds comprehensive TRACES.md, a generated traces JSON Schema, and a script to regenerate trace docs.

• Tests

  • Adds extensive unit, UI (compile-fail), and integration tests validating macros, schema validation, and trace recording.

• Chores

  • CI/workflow updates, workspace/manifest and toolchain adjustments, line-ending rule, and build/test command standardization.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Adds a new amaru-observability subsystem (proc‑macros, runtime registry, schemas), migrates many crates to schema-backed tracing (#[trace], trace_span!, trace_record!), updates CI and toolchain to validate/generate TRACES.md, and bumps trace payload shapes and tests.

Changes

Cohort / File(s)	Summary
Workspace & CI Cargo.toml, .github/workflows/continuous-integration.yml, Makefile, .cargo/config.toml, .gitattributes, rust-toolchain.toml, crates/*/Cargo.toml	Adds amaru-observability to workspace, updates CI to run schema validation and TRACES.md checks, changes test invocations to cargo test --workspace --all-targets, enforces LF for **/*.stderr, and adds rust-src toolchain component.
Observability crate & runtime crates/amaru-observability/Cargo.toml, crates/amaru-observability/src/*, crates/amaru-observability/src/registry.rs, crates/amaru-observability/src/schemas.rs, crates/amaru-observability/examples/*	New crate exposing macros, a runtime SchemaEntry registry (inventory), schema definitions, examples, and CLI/runtime APIs for dumping schemas.
Procedural macros crates/amaru-observability/macros/Cargo.toml, crates/amaru-observability/macros/src/{lib.rs,define_schemas.rs,traces.rs,utils.rs}, crates/amaru-observability/macros/*	New proc‑macro crate implementing define_schemas!, #[trace], trace_record!, trace_span!, schema parsing, validators, codegen, tests and trybuild UI tests.
Instrumentation migration crates/*/{consensus,ledger,network,stores,protocols,pure-stage,simulation}/src/...	Replaces many tracing::instrument / string spans with schema-anchored #[trace(...)], trace_span!, constants; updates imports and a few signatures (e.g., pull stage adds eff param) and removes ChainSyncEvent::set_span.
Trace schema assets & docs crates/amaru-observability/src/schemas.rs, docs/TRACES.md, docs/traces-schema.json, scripts/generate-traces-doc	Adds full trace schema DSL output, JSON Schema asset, docs generator script, and generated TRACES.md content.
CLI wiring crates/amaru/src/bin/amaru/cmd/dump_schemas.rs, crates/amaru/src/bin/amaru/main.rs, crates/amaru/src/bin/amaru/cmd/mod.rs	New dump-schemas CLI command and wiring to emit JSON Schema from runtime registry.
Tests & fixtures crates/amaru-observability/macros/tests/*, crates/amaru-observability/tests/*, crates/amaru-ledger/tests/data/**/expected.traces, simulation/.../tests/traces.rs	Adds many macro unit/UI tests and updates expected-trace fixtures to flattened field shapes and new span names.
Collector behavior crates/amaru-tracing-json/src/lib.rs	Collector now accumulates fields on_record and emits the full span JSON on close (instead of on_enter).
Docs & EDR engineering-decision-records/007-observability.md, monitoring/README.md	Expands EDR with tracing-schema design; simplifies monitoring docs and redirects detailed lists to generated TRACES.md.

Sequence Diagram(s)

sequenceDiagram
    participant Dev as Developer
    participant Macro as ProcMacro (amaru-observability-macros)
    participant Registry as Schema Registry (inventory)
    participant Generated as Generated Wrapper
    participant Runtime as App / tracing_subscriber

    Dev->>Macro: annotate fn with #[trace(path, ...)]
    Macro->>Registry: resolve schema path & metadata
    alt invalid schema
        Registry-->>Macro: not found / validation error
        Macro-->>Dev: emit compile_error!
    else valid schema
        Macro->>Generated: emit validators + RECORD/INSTRUMENT wrappers
        Generated-->>Dev: expanded instrumented function
        Dev->>Runtime: run instrumented function
        Runtime->>Runtime: create span, record fields (on_record)
        Runtime->>Runtime: close span -> emit final JSON
    end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• feat: improve naming for consensus logs and spans #549 — broad consensus-stage tracing refactor; likely conflicts on span/name changes.
• feat: add a span for the pull stage #533 — overlaps pull stage instrumentation and the pull.rs signature change (Effects param).
• reinstate tracing spans for stages #412 — touches consensus stage tracing (receive_header, select_chain, validate_*), overlapping instrumentation edits.

Suggested reviewers

• abailly
• rkuhn
• etorreborre

Poem

🌿 A schema sprouted in code and light,
Macros hum, spans take flight,
Fields flattened, docs parade,
CI guards the trace cascade,
Cheers — observability’s looking right.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat: improve observability' directly aligns with the primary objective of this PR, which introduces a comprehensive observability infrastructure overhaul including the amaru-observability crate and schema-driven tracing.
Linked Issues check	✅ Passed	The PR fully implements all primary objectives from issue #640: refined trace documentation, comprehensive schema definitions, reworked tracing across codebase, EDR captured, tooling created, and CI integration for documentation generation and regression detection.
Out of Scope Changes check	✅ Passed	All changes are tightly scoped to observability improvements. File modifications consistently involve replacing instrumentation attributes, adding amaru-observability dependencies, updating trace definitions, and generating documentation—all directly supporting the PR's observability objectives.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch jeluard/observability

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[etorreborre]

This is a very nice feature to standardize our traces! We should not wait too much to merge it because there are going to be some merge conflicts and it will be good to have this kind of support when the consensus stages are being reworked.

Maybe the biggest thing that could be missing is the ability to annotate a span as "internal" so that it can still be used but not part of the external API.

[etorreborre]

I'm just leaving this as a note.

I think that we need to use the levels a bit better and not leave everything at the TRACE level.
For example, for the amaru::consensus::chain_sync target, the decode_header span should probably be at the DEBUG level, pull at the TRACE level (this is a low-level mechanism) and maybe select_chain at the INFO level.

This way we can use the INFO level to see high-level normal processing, and other levels to drill-down.
The important thing IMO is to have the ability to drill-down target by target, and for example see the possible interactions between the ledger and the consensus by selecting amaru_ledger=debug, amaru_consensus=info.

[jeluard]

I generally agree although I feel like it mostly apply to human logs, not computer traces.
For computer traces we probably want everything (and filtering some actually breaks parent/child relationships). At least for OTEL.
Now we also have an issue of too many logs generated for now. Especially when the node catches up and apply lots of blocks.