Merge pull request #352 from pragma-org/stevan/sim-readme

stevana · web-flow · commit ee00cbe68532 · 2025-07-30T15:24:11.000+02:00
feat/simulation readme
diff --git a/simulation/amaru-sim/README.md b/simulation/amaru-sim/README.md
@@ -1,23 +1,109 @@
 # Amaru Simulator
 
-This component aims at implementing a _Simulator_ for the Ouroboros Consensus, in Rust, using Amaru components. The main goal of this work is to be able to test the consensus part as deeply as possible, using different strategies, in increasing order of fidelity:
+This component aims at implementing a _Simulator_ for the Ouroboros Consensus,
+in Rust, using Amaru components. The main goal of this work is to be able to
+test the consensus part as deeply as possible, using different strategies, in
+increasing order of fidelity:
 
-1. ✅ In-process deterministic testing, completely simulating the environment, allowing arbitrary fault injections and full control over concurrency and other side-effects
-2. ✅ [Maelstrom](https://github.com/jepsen-io/maelstrom/)-like testing through stdin/stdout interface ignoring network interactions
-3. 🔴 [Jepsen](https://github.com/jepsen-io/jepsen)-like testing through full-blown deployment of a cluster and actual networking stack
+1. ✅ In-process deterministic testing, completely simulating the environment,
+   allowing arbitrary fault injections and full control over concurrency and
+   other side-effects
+2. ✅ [Maelstrom](https://github.com/jepsen-io/maelstrom/)-like testing through
+   stdin/stdout interface ignoring network interactions
+3. 🔴 [Jepsen](https://github.com/jepsen-io/jepsen)-like testing through
+   full-blown deployment of a cluster and actual networking stack
 4. 🔴 [Antithesis](https://antithesis.com) support
 
+## Overview
+
+The main components of the simulator are:
+
+* Test case generation, found in
+  [`src/simulator/generate.rs`](src/simulator/generate.rs), which uses the
+  pre-generated block tree which is saved in
+  [`tests/data/chain.json`](tests/data/chain.json);
+* The (discrete-event) simulator itself, lives in
+  [`src/simulator/simulate.rs`](src/simulator/simulate.rs);
+* The property-based test and property that uses the simulator, defined in
+  [`src/simulator/mod.rs`](src/simulator/mod.rs);
+  and property that uses the simulator is defined;
+* The actual Rust `#test` which gets picked up by `cargo test`, found in
+  [`tests/simulation.rs`](tests/simulation.rs). 
+
 ## Usage
 
-The `simulator` executable is a pared-down version of Amaru where network communications are abstracted away. It's packaged as a test so running it amounts to:
+The `simulator` test is a pared-down version of Amaru where network
+communications are abstracted away.
+
+The test can be run as follows (environment variables can be used to override
+options, we show the default values here):
 
 ```
-cargo test -p amaru-sim
+AMARU_NUMBER_OF_TESTS=50             # Set the number of test cases to generate. \
+AMARU_NUMBER_OF_NODES=1              # Set the number of nodes in a simulation. \
+AMARU_NUMBER_OF_UPSTREAM_PEERS=2     # Set the number of upstream peers.
+AMARU_DISABLE_SHRINKING=0            # Set to 1 to disable shrinking. \
+AMARU_TEST_SEED=                     # Seed to use to reproduce a test case. \
+AMARU_PERSIST_ON_SUCCESS=0           # Set to 1 to persist pure-stage schedule on success. \
+AMARU_SIMULATION_LOG="error"         # Only show error-level logging. \
+\
+cargo test run_simulator
 ```
 
-By default, it only logs `error` level and above log entries, but one filter logs by setting the `AMARU_SIMULATION_LOG` environment variable to an appropriate value.
+## Debugging failures
+
+When the test fails, the output looks something like this:
+
+```
+ Minimised input (0 shrinks):
+
+  Envelope { src: "c1", dest: "n1", body: Fwd { msg_id: 0, slot: Slot(31), hash: "2487bd", header: "828a0118" } }
+  Envelope { src: "c1", dest: "n1", body: Fwd { msg_id: 1, slot: Slot(38), hash: "4fcd1d", header: "828a0218" } }
+  Envelope { src: "c1", dest: "n1", body: Fwd { msg_id: 2, slot: Slot(41), hash: "739307", header: "828a0318" } }
+  Envelope { src: "c1", dest: "n1", body: Fwd { msg_id: 3, slot: Slot(55), hash: "726ef3", header: "828a0418" } }
+  Envelope { src: "c1", dest: "n1", body: Fwd { msg_id: 4, slot: Slot(93), hash: "597ea6", header: "828a0518" } }
+  [...]
+
+History:
+
+    0.  "c1" ==> "n1"   Fwd { msg_id: 0, slot: Slot(31), hash: "2487bd", header: "828a0118" }
+    1.  "n1" ==> "c1"   Fwd { msg_id: 0, slot: Slot(31), hash: "2487bd", header: "828a0118" }
+    2.  "c2" ==> "n1"   Fwd { msg_id: 0, slot: Slot(31), hash: "2487bd", header: "828a0118" }
+    3.  "c2" ==> "n1"   Fwd { msg_id: 1, slot: Slot(38), hash: "4fcd1d", header: "828a0218" }
+    4.  "n1" ==> "c2"   Fwd { msg_id: 1, slot: Slot(38), hash: "4fcd1d", header: "828a0218" }
+    [...]
+
+Error message:
+
+  tip of chains don't match, expected:
+    (Bytes { bytes: "fcb4a51..." }, Slot(990))
+  got:
+    (Bytes { bytes: "gcb4a51..." }, Slot(990))
+
+Saved schedule: "./failure-1752489042.schedule"
+
+Seed: 42
+```
+
+Let's break the components down:
+
+* The minimised failing test case is printed so that it can be copy-pasted into a
+  `#test` to create a regression test;
+* The history is the same as test case, but the `src` and `dest` of each
+  message has been pretty printed to be easier to read and it also contains the
+  responses that we got back from the system under test. The history is what the
+  property is checked against;
+* The error message is how the property failed;
+* Saved schedule is the execution schedule from `pure-stage`, which can provide
+  low-level details about how the stage processing happened. See the following
+  [note](https://github.com/pragma-org/amaru/wiki/log-::-2025%E2%80%9006#debugging-simulation-tests-failure)
+  for more details of how to use this information;
+* The seed is what was used to produce the test case, it can be used to replay
+  the test (see `AMARU_TEST_SEED` above).
 
 ## References
 
-* [Cardano Consensus and Storage Layer](https://ouroboros-consensus.cardano.intersectmbo.org/assets/files/report-b72e7d765cfee85b26dc035c52c6de84.pdf)
-* [Ouroboros Network Specification](https://ouroboros-network.cardano.intersectmbo.org/pdfs/network-spec/network-spec.pdf)
+* [Cardano Consensus and Storage
+  Layer](https://ouroboros-consensus.cardano.intersectmbo.org/assets/files/report-b72e7d765cfee85b26dc035c52c6de84.pdf)
+* [Ouroboros Network
+  Specification](https://ouroboros-network.cardano.intersectmbo.org/pdfs/network-spec/network-spec.pdf)
diff --git a/simulation/amaru-sim/src/simulator/mod.rs b/simulation/amaru-sim/src/simulator/mod.rs
@@ -95,6 +95,9 @@ pub struct Args {
     #[arg(long, default_value = "2")]
     pub number_of_upstream_peers: Option<u8>,
 
+    #[arg(long)]
+    pub disable_shrinking: bool,
+
     /// Seed for simulation testing.
     #[arg(long)]
     pub seed: Option<u64>,
@@ -352,6 +355,7 @@ pub fn run(rt: tokio::runtime::Runtime, args: Args) {
     let number_of_tests = args.number_of_tests.unwrap_or(50);
     let number_of_nodes = args.number_of_nodes.unwrap_or(1);
     let number_of_upstream_peers = args.number_of_upstream_peers.unwrap_or(2);
+    let disable_shrinking = args.disable_shrinking;
     let trace_buffer = Arc::new(parking_lot::Mutex::new(TraceBuffer::new(42, 1_000_000_000)));
 
     let spawn = || {
@@ -371,6 +375,7 @@ pub fn run(rt: tokio::runtime::Runtime, args: Args) {
             number_of_tests,
             seed,
             number_of_nodes,
+            disable_shrinking,
         },
         spawn,
         generate_entries(
@@ -405,7 +410,7 @@ fn chain_property(
                             .expect("empty chain data");
                         if actual != expected {
                             return Err(format!(
-                                "tip of chains don't match, expected {:?}, got {:?}",
+                                "tip of chains don't match, expected:\n    {:?}\n  got:\n    {:?}",
                                 expected, actual
                             ));
                         }
diff --git a/simulation/amaru-sim/src/simulator/simulate.rs b/simulation/amaru-sim/src/simulator/simulate.rs
@@ -54,6 +54,7 @@ pub struct SimulateConfig {
     pub number_of_tests: u32,
     pub seed: u64,
     pub number_of_nodes: u8,
+    pub disable_shrinking: bool,
 }
 
 #[derive(Debug, Clone, PartialEq, Serialize)]
@@ -289,23 +290,34 @@ pub fn simulate<Msg, F>(
     for test_number in 1..=config.number_of_tests {
         let entries: Vec<Reverse<Entry<Msg>>> = generator(&mut rng);
 
-        match run_test(config.number_of_nodes, &spawn, &property)(&entries) {
-            (_history, Err(reason)) => {
-                let (shrunk_entries, (shrunk_history, result), number_of_shrinks) = shrink(
-                    run_test(config.number_of_nodes, &spawn, &property),
-                    entries,
-                    |result| result.1 == Err(reason.clone()),
-                );
-                assert_eq!(Err(reason.clone()), result);
-                display_failure(
-                    test_number,
-                    config.seed,
-                    shrunk_entries,
-                    number_of_shrinks,
-                    shrunk_history,
-                    trace_buffer.clone(),
-                    reason,
-                );
+        let test = run_test(config.number_of_nodes, &spawn, &property);
+        match test(&entries) {
+            (history, Err(reason)) => {
+                if config.disable_shrinking {
+                    let number_of_shrinks = 0;
+                    display_failure(
+                        test_number,
+                        config.seed,
+                        entries,
+                        number_of_shrinks,
+                        history,
+                        trace_buffer.clone(),
+                        reason,
+                    );
+                } else {
+                    let (shrunk_entries, (shrunk_history, result), number_of_shrinks) =
+                        shrink(test, entries, |result| result.1 == Err(reason.clone()));
+                    assert_eq!(Err(reason.clone()), result);
+                    display_failure(
+                        test_number,
+                        config.seed,
+                        shrunk_entries,
+                        number_of_shrinks,
+                        shrunk_history,
+                        trace_buffer.clone(),
+                        reason,
+                    );
+                }
                 break;
             }
             (_history, Ok(())) => continue,
@@ -348,7 +360,7 @@ fn display_failure<Msg: Debug>(
                 Minimised input ({number_of_shrinks} shrinks):\n\n{}\n \
                 History:\n\n{}\n \
                 Error message:\n\n  {}\n\n \
-                {} \
+                {}\n \
                 Seed: {}\n",
             test_case,
             history_string,
@@ -496,6 +508,7 @@ mod tests {
                 number_of_tests,
                 seed,
                 number_of_nodes,
+                disable_shrinking: false,
             },
             spawn,
             generator,
@@ -566,6 +579,7 @@ mod tests {
                 number_of_tests,
                 seed,
                 number_of_nodes,
+                disable_shrinking: false,
             },
             spawn,
             generate_messages,
diff --git a/simulation/amaru-sim/tests/simulation.rs b/simulation/amaru-sim/tests/simulation.rs
@@ -12,13 +12,23 @@ fn run_simulator() {
         chain_dir: "./chain.db".into(),
         block_tree_file: "tests/data/chain.json".into(),
         start_header: Hash::from([0; 32]),
-        number_of_tests: Some(50),
-        number_of_nodes: Some(1),
-        number_of_upstream_peers: Some(2),
+        number_of_tests: env::var("AMARU_NUMBER_OF_TESTS")
+            .ok()
+            .and_then(|v| v.parse::<u32>().ok())
+            .or(Some(50)),
+        number_of_nodes: env::var("AMARU_NUMBER_OF_NODES")
+            .ok()
+            .and_then(|v| v.parse::<u8>().ok())
+            .or(Some(1)),
+        number_of_upstream_peers: env::var("AMARU_NUMBER_OF_UPSTREAM_PEERS")
+            .ok()
+            .and_then(|v| v.parse::<u8>().ok())
+            .or(Some(2)),
+        disable_shrinking: std::env::var("AMARU_DISABLE_SHRINKING").is_ok_and(|v| v == "1"),
         seed: std::env::var("AMARU_TEST_SEED")
             .ok()
             .and_then(|s| s.parse().ok()),
-        persist_on_success: std::env::var("AMARU_PERSIST_ON_SUCCESS").is_ok(),
+        persist_on_success: std::env::var("AMARU_PERSIST_ON_SUCCESS").is_ok_and(|v| v == "1"),
     };
 
     tracing_subscriber::fmt()
