Skip to content

Commit 58b9468

Browse files
dsarnoclaude
andauthored
docs(stormtest): document Windows reload-churn limitations + pathing (#512)
Reads/writes concurrency is platform-agnostic and works on Windows, but reload churn (SS_RELOAD=1, the default) currently does not. Against a plugin-managed server the harness wedges past CALL_TIMEOUT when the server is killed mid-reload under load (#513); the documented "run the server externally" mitigation also fails on Windows because serve-this-worktree is bash-only and a hand-started --reload server gets killed by the reload (#514). The editor itself survives reloads. Also documents the POSIX->Windows invocation differences (venv path, $TMPDIR -> %TEMP%). Resilience tracked in #509. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 5fe08b8 commit 58b9468

2 files changed

Lines changed: 39 additions & 0 deletions

File tree

AGENTS.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -288,6 +288,8 @@ SS_URL=http://127.0.0.1:8010/mcp .venv/bin/python script/stormtest.py # target
288288

289289
To stress a *branch's* code (plugin + server), point a Godot editor at that worktree's `test_project/` and serve its `src/` via `script/serve-this-worktree` (external server, so `editor_reload_plugin` exercises reload without killing the server), then run stormtest against it. A full JSON snapshot lands in `$TMPDIR/stormtest_report.json` (override with `SS_REPORT`), flushed every few seconds so a crash/kill still leaves data. A small `EDITOR_NOT_READY` / `NODE_NOT_FOUND` / `CONNECTION` error rate is expected noise under concurrency + reloads — watch instead for the process dying, a reload that never recovers, or one op with pathological error/latency.
290290

291+
On Windows, a reads-dominant run (`SS_RELOAD=0`) works, but **reload churn (`SS_RELOAD=1`, the default) currently wedges the harness** and the external-server mitigation doesn't apply (`serve-this-worktree` is bash-only; a hand-started external `--reload` server gets killed by the reload). The editor itself survives reloads — only the harness hangs. Use `.venv\Scripts\python.exe` and note `$TMPDIR` is `%TEMP%`. See the "Windows / cross-platform notes" callout in `docs/STRESS_TESTING.md` (issues #513 / #514; resilience tracked in #509).
292+
291293
**Guardrails built into the test runner:**
292294
- **Zero-assertion detection**: Tests that complete with 0 assertions are flagged as failures ("Test completed with 0 assertions — likely skipped its logic"). This catches tests that silently `return` before asserting anything.
293295
- **Resilient discovery**: If a `.gd` file fails to load (parse error, duplicate method, wrong base class), the rest of the suites still run and the failing files are reported in `load_errors`.

docs/STRESS_TESTING.md

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,43 @@ SS_RELOAD=0 SS_WORKERS=4 SS_WAVES=3 .venv/bin/python script/stormtest.py
6363
SS_URL=http://127.0.0.1:8010/mcp .venv/bin/python script/stormtest.py
6464
```
6565

66+
### Windows / cross-platform notes
67+
68+
> ⚠️ **Running on Windows? Reads/writes work; reload churn does not (yet).**
69+
>
70+
> **Concurrent reads/writes are fine.** The harness *logic* is platform-agnostic:
71+
> the in-editor scratch paths (`res://_stormtest/…`) use Godot's virtual
72+
> filesystem, and the report path goes through `tempfile.gettempdir()` /
73+
> `os.path.join`, so both resolve correctly on every OS. A reads-dominant run
74+
> (`SS_RELOAD=0`) is clean on Windows.
75+
>
76+
> **Reload churn (`SS_RELOAD=1`, the default) currently does NOT work on Windows**
77+
> — two independent problems, both tracked:
78+
> - Against a plugin-managed server, the first `editor_reload_plugin` **wedges the
79+
> harness**: the asyncio loop stalls past `CALL_TIMEOUT` when the server is
80+
> killed mid-reload under concurrent load. The *editor* survives fine (it
81+
> reloads and re-registers a new session) — only the harness hangs. See
82+
> [#513](https://github.com/hi-godot/godot-ai/issues/513).
83+
> - The "run the server externally" mitigation **also fails on Windows**:
84+
> `script/serve-this-worktree` is bash-only (no PowerShell port, and it never
85+
> passes `--ws-port`), and even a hand-started external `--reload` server gets
86+
> **killed by the reload** (`_stop_server` takes down the port owner with no
87+
> respawn). See [#514](https://github.com/hi-godot/godot-ai/issues/514).
88+
>
89+
> Until those land, validate reload survival on Windows with a *single-threaded*
90+
> reload loop (reload → reconnect → confirm a new `session_id`) rather than the
91+
> concurrent churn mode.
92+
>
93+
> **Invocation also differs (POSIX → Windows):**
94+
> - **venv interpreter** — use `.venv\Scripts\python.exe`, not `.venv/bin/python`.
95+
> - **`$TMPDIR`** in the examples is POSIX; the report lands in the platform temp
96+
> dir (`%TEMP%` on Windows). Pass an explicit `SS_REPORT=…` for a known
97+
> location, and prefer forward slashes (Python accepts them on Windows and they
98+
> dodge backslash-escaping surprises).
99+
>
100+
> Making the tooling resilient enough to drop this heads-up is tracked in
101+
> [#509](https://github.com/hi-godot/godot-ai/issues/509).
102+
66103
### Knobs (env)
67104

68105
| Var | Default | Meaning |

0 commit comments

Comments
 (0)