Skip to content

Add --shuffle option to randomize test execution order #530

@tclune

Description

@tclune

Feature Request

Add command-line options to randomize the order in which tests are executed within each test suite.

Motivation

Randomizing test execution order helps detect hidden dependencies between tests and ensures tests are properly isolated.

Proposed Implementation

Command-line options:

  • --shuffle - Enable random test ordering within each suite
  • --seed=N - Specify random seed for reproducibility (implies --shuffle, 0=time-based)

Behavior:

  • Shuffle tests within each suite independently (preserve suite boundaries)
  • Use time-based random seed if --shuffle specified without --seed
  • Tests in nested suites maintain their suite grouping

Implementation plan:

  1. Add CLI arguments in src/funit/FUnit.F90
  2. Implement shuffle logic in src/funit/core/TestSuite.F90
  3. Add unit tests in tests/funit-core/Test_TestSuite_Shuffle.F90
  4. Add integration test (ctest with shell script)

Testing

  • Unit tests to verify shuffle behavior, seed reproducibility, suite boundary preservation
  • Integration tests via command-line execution

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions