Tighten 7 tests that passed without verifying behavior#458
Merged
Conversation
Audit of the GDScript + Python test suites surfaced seven tests whose assertions didn't actually pin down the behavior they claimed to test — they would have stayed green under regressions in the code paths they exercise. GDScript (test_project/tests/): - test_theme.gd::test_theme_set_color_accepts_dict only checked the response envelope. Now reloads the theme and asserts the stored value is a Color with the right channels, so a missing dict->Color coercion can't pass. - test_theme.gd::test_create_theme_overwrite_allowed and test_material.gd::test_create_overwrite_allowed now assert overwritten=true on the response and that the target file still exists, instead of just confirming the call returned data. - test_material.gd::test_set_param_bool now reads emission_enabled back off the loaded material (matching the read-back pattern its float and color siblings already use). - test_animation.gd::test_add_property_track_transition_raw_float now reads the stored keyframes and asserts both that the dict values coerced to Vector3 and that the raw-float transition landed on each key — the prior version would pass even if either silently no-oped. Python (tests/unit/): - test_runtime_handlers.py::test_audio_player_create_blocks_when_not_writable was catching bare Exception, which would mask unrelated bugs. Narrowed to GodotCommandError to match the surrounding readiness-gate tests. - test_self_update_smoke_harness.py::test_v240_preflight_passes_when_both_files_present had zero assertions — it called the validator and returned. Added an assertion that the call returns None so the runner's zero-assertion guard has something to verify and the intent is documented. https://claude.ai/code/session_01H8CLhwuNUA4NoEZ4neJk78
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Linux + macOS Godot tests passed on the previous run; Windows failed in ~89s (vs ~134s on green main-branch runs), suggesting either a flake or an early-pipeline failure. Pushing this empty commit to disambiguate. https://claude.ai/code/session_01H8CLhwuNUA4NoEZ4neJk78
Contributor
There was a problem hiding this comment.
Pull request overview
Improves the reliability of the test suite by strengthening 7 tests (Python + GDScript) that previously could pass without actually validating the intended behavior, aligning assertions with real persisted/round-tripped outcomes where applicable.
Changes:
- Add an explicit assertion to prevent a zero-assertion Python unit test from silently passing.
- Narrow a Python exception expectation from a broad
ExceptiontoGodotCommandErrorfor readiness-gated behavior. - Strengthen several GDScript tests by reading back stored resources/values and asserting the actual persisted (or runtime-stored) state, plus overwrite metadata.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_self_update_smoke_harness.py | Adds an assertion to ensure the preflight validator test actually verifies success. |
| tests/unit/test_runtime_handlers.py | Narrows expected exception type for a non-writable-session gate to GodotCommandError. |
| test_project/tests/test_theme.gd | Tightens overwrite and dict→Color behavior checks via additional assertions and resource read-back. |
| test_project/tests/test_material.gd | Tightens overwrite and bool parameter persistence checks via resource read-back. |
| test_project/tests/test_animation.gd | Verifies stored keyframe values and per-key transitions after adding a property track. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- test_animation.gd: add value assert for v0 so both keyframes carry the same depth of check (was: only v1 had a value assert; v0 was type-only). - test_theme.gd: assert the alpha channel in test_theme_set_color_accepts_dict — the test dict explicitly passes a=1.0, so leaving it unchecked left a gap in the dict→Color coercion coverage. https://claude.ai/code/session_01H8CLhwuNUA4NoEZ4neJk78
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Audit of the test suites (Python + GDScript) surfaced 7 tests whose assertions did not actually pin down the behavior they claimed to test — they would have stayed green under regressions in the code paths they exercise. Five matched the "counts instead of stored Variants" /
assert_has_keyweakness called out inCLAUDE.md's test-hygiene checklist; two were Python-side cousins (overly broadpytest.raises(Exception)and a zero-assertion test).GDScript fixes
test_theme.gd::test_theme_set_color_accepts_dict— only checked the response envelope. Now reloads the theme and asserts the stored value is aColorwith the right channels, so a missing dict→Color coercion can't pass.test_theme.gd::test_create_theme_overwrite_allowedandtest_material.gd::test_create_overwrite_allowed— now assertoverwritten=trueon the response and that the target file still exists, instead of just confirming the call returneddata.test_material.gd::test_set_param_bool— now readsemission_enabledback off the loaded material (matching the read-back pattern its float and color siblings already use).test_animation.gd::test_add_property_track_transition_raw_float— now reads the stored keyframes and asserts both that the dict values coerced toVector3and that the raw-floattransitionlanded on each key. The prior version would pass even if either silently no-oped.Python fixes
test_runtime_handlers.py::test_audio_player_create_blocks_when_not_writable— was catching bareException, which would mask unrelated bugs. Narrowed toGodotCommandErrorto match the surrounding readiness-gate tests.test_self_update_smoke_harness.py::test_v240_preflight_passes_when_both_files_present— had zero assertions; it called the validator and returned. Addedassert smoke._require_v240_plus_addon_shape(...) is Noneso the runner's zero-assertion guard has something to verify and the intent is documented.Notes — wider patterns observed (not fixed here)
assert_is_error(result)across the GDScript suite omit the expected error code (vs ~153 that supply one). That accepts any error as success, including unrelated bugs. Worth fixing in tests whose names claim a specific failure (e.g.test_create_node_non_node_type) but is too large to bundle into this PR.editor_undo(_undo_redo)calls flagged attest_node.gd:283, 298, 928are cleanup-only undos with no post-undo assertion, so they don't violate the CLAUDE.md rule. Left as-is.test_ui.gd:524/551/575/607use the non-_overridetheme getters but are gated onhas_theme_*_override(...)first, which guards against the silent-fallback failure mode. Godot 4.6 dropped the_overridegetter variants, per the inline comment. Left as-is.Test plan
ruff check tests/unit/test_runtime_handlers.py tests/unit/test_self_update_smoke_harness.py— cleanpytest tests/unit/test_runtime_handlers.py tests/unit/test_self_update_smoke_harness.py -q— 263 passedtest_runvia MCP) — to be exercised when CI picks up the branch / a reviewer runs the editor; the edits are syntactic-mirror copies of patterns already used elsewhere in the same fileshttps://claude.ai/code/session_01H8CLhwuNUA4NoEZ4neJk78
Generated by Claude Code