feat(ci): implement nextest and optimize Docker test builds by gustavovalverde · Pull Request #9435 · ZcashFoundation/zebra

gustavovalverde · 2025-04-17T08:26:45Z

Motivation

Zebra's CI/CD pipeline had several inefficiencies that were impacting build and test execution times:

Fragmented Test Execution: Individual environment variables were used to control different test suites (RUN_ALL_TESTS, STATE_FAKE_ACTIVATION_HEIGHTS, SYNC_LARGE_CHECKPOINTS_EMPTY, etc.), sometimes in combination with feature gates, leading to inconsistent test configuration and complex workflow management.
Feature Gate Inefficiencies: Different feature combinations between build-time and runtime were causing Rust to rebuild artifacts unnecessarily, slowing down the build process.
Complex Entrypoint Logic: The entrypoint.sh script contained complex conditional logic for different test scenarios, making it harder to maintain and understand.
Verbose CI Workflows: Each test type required separate environment variables or feature gates configurations across multiple workflow files.

Fixes: #9331

Solution

This PR implements a modernization of Zebra's test execution and Docker build system:

1. Nextest Integration with Centralized Configuration

Added .config/nextest.toml with 17 specialized test profiles covering all test scenarios:
- all-tests: Runs all tests except dependency checks
- Individual profiles for each test type: sync-full-mainnet, lwd-grpc-wallet, rpc-submit-block, etc.
- Proper timeout configurations and success output settings per test type
Replaced most environment variable-based test control with a single NEXTEST_PROFILE variable
Centralized test filtering and scoping logic from scattered shell scripts into declarative configuration

2. Docker Build Optimization

Streamlined feature sets: Uses minimal features (default-release-binaries proptest-impl lightwalletd-grpc-tests zebra-checkpoints) for testing to prevent unnecessary rebuilds
Simplified test stage: Reduced Docker commands and improved layer caching
Enhanced cargo nextest integration: Added automatic nextest binary installation and optimized build process
Updated .dockerignore to include .config directory for nextest configuration

3. CI/CD Workflow Modernization

Updated 2 major workflow files (sub-ci-unit-tests-docker.yml, sub-ci-integration-tests-gcp.yml) with 15+ test jobs converted to use nextest profiles
Simplified environment variable management: Replaced dozens of individual test flags with unified NEXTEST_PROFILE usage
Improved log streaming: Simplified deployment test monitoring by removing complex grep patterns and relying on container exit codes
Enhanced error handling: More reliable test result detection using container exit status

4. Entrypoint Simplification (Phase 1)

Reduced entrypoint.sh complexity: Moved test execution logic to nextest profiles
Added nextest integration: When NEXTEST_PROFILE is set, the entrypoint uses cargo nextest run with appropriate flags
Maintained backward compatibility: Existing entry points still work while new nextest path is preferred

5. Test Execution Improvements

Faster test execution: Nextest's parallel execution and smart filtering reduce test times
Consistent timeout handling: Proper timeout configurations per test type prevent false failures
Improved test output: Better progress reporting and immediate success output for long-running tests

Performance Impact

Based on CI run comparisons:

Significant reduction in test execution times across all test suites
Eliminated unnecessary Rust rebuilds caused by feature flag mismatches
Streamlined CI workflow execution with unified environment variable approach

Migration Path

Backward Compatible: Existing entrypoint.sh functionality preserved
Gradual Adoption: Docker tests automatically use nextest, existing tests continue to work
Future-Ready: Foundation laid for further optimizations (test sharding, more granular grouping)

Testing

All existing test suites pass with nextest profiles
Docker builds complete successfully with optimized features
CI workflows execute correctly with new environment variables

- Add feature gates to lightwalletd test infrastructure to prevent compilation errors when lightwalletd-grpc-tests is disabled - Add feature gate to indexer test to prevent compilation errors when indexer is disabled - Move lightwalletd-related imports and constants behind feature gates - Wrap gRPC code generation in feature-conditional module - Fix GitHub Actions workflow to pass features as single string - Restore missing lightwalletd_failure_messages method with feature gate - Add missing DATABASE_FORMAT_UPGRADE_IS_LONG import This ensures --no-default-features builds work correctly while maintaining full functionality when features are enabled.

- Remove unused DATABASE_FORMAT_UPGRADE_IS_LONG import - Update references to use common::cached_state::DATABASE_FORMAT_UPGRADE_IS_LONG - Fix unused import linting warning

This commit addresses several issues related to running tests within Docker and CI environments. The initial problem was a permissions error where `nextest` could not write to its store directory. This was resolved by adjusting `CARGO_HOME` and `CARGO_TARGET_DIR` to be relative to the user's home directory within the Docker image. A subsequent issue was discovered where test filters in `nextest.toml` were platform-specific, causing the entire test suite to run on `x86_64` CI runners, leading to failures. The configuration has been refactored to be platform-agnostic, ensuring filters are applied correctly on all architectures. Additionally, the `Dockerfile` has been updated to use a multi-stage build for fetching the `lightwalletd` binary, resolving multi-platform build failures. The test entrypoint script was also improved to correctly handle ignored tests and provide cleaner logs. Finally, the GCP integration test workflow has been simplified to rely on the container's exit code for determining test success, removing fragile log parsing.

.github/workflows/sub-ci-unit-tests-docker.yml

conradoplg

Looks good, added some minor suggestions.

I'll wait before approving in order to check if we want to do the next release before merging this.

zebrad/tests/acceptance.rs

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

conradoplg

Looks good, thanks!

conradoplg · 2025-08-11T16:45:10Z

@Mergifyio requeue

mergify · 2025-08-11T16:45:17Z

requeue

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

gustavovalverde · 2025-08-11T18:16:53Z

I'm admin merging to clean-up the merge message

gustavovalverde temporarily deployed to dev April 17, 2025 08:26 — with GitHub Actions Inactive

gustavovalverde had a problem deploying to dev April 17, 2025 08:26 — with GitHub Actions Failure

gustavovalverde mentioned this pull request Apr 17, 2025

fix(CI): Cache compilation results #9328

Closed

5 tasks

gustavovalverde changed the title ~~ref(docker): improve cargo caching by aligning mounts with CARGO_HOME~~ ref(docker): improve caching by aligning mounts with CARGO_HOME Apr 17, 2025

gustavovalverde temporarily deployed to dev April 17, 2025 08:27 — with GitHub Actions Inactive

gustavovalverde had a problem deploying to dev April 29, 2025 09:40 — with GitHub Actions Failure

gustavovalverde temporarily deployed to dev April 29, 2025 09:40 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev April 29, 2025 09:41 — with GitHub Actions Inactive

gustavovalverde changed the title ~~ref(docker): improve caching by aligning mounts with CARGO_HOME~~ ref(docker): improve cache by aligning mounts with CARGO_HOME May 5, 2025

gustavovalverde temporarily deployed to dev May 5, 2025 08:01 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev May 5, 2025 08:25 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev May 5, 2025 08:26 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev May 5, 2025 08:53 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev May 5, 2025 08:54 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev May 5, 2025 09:09 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev May 5, 2025 09:10 — with GitHub Actions Inactive

gustavovalverde added 17 commits August 4, 2025 13:38

feat: add indexer as a default feature

a5e10c9

refactor(tests): remove unused import and use fully qualified path

4a0755f

- Remove unused DATABASE_FORMAT_UPGRADE_IS_LONG import - Update references to use common::cached_state::DATABASE_FORMAT_UPGRADE_IS_LONG - Fix unused import linting warning

refactor(docker): simplify dockerfile

93311d0

fix: formatting

ed0607c

fix: use the default cargo directories

d7e72e0

fix: build

f1cd560

fix: cargo ownership

e88ddf0

fix: use correct test filtering with nextest

6da21f1

fix: avoid cargo cache invalidation

74a2e3e

fix: improve caching

ca325dc

fix: segregate sync-past-mandatory test

b844371

fix: show output for specific test

06efc95

revert: most sync and lwd changes

d99090e

fix: remove indexer

dd9aeaf

chore: reduce diff

45cf591

github-advanced-security bot found potential problems Aug 5, 2025

View reviewed changes

.github/workflows/sub-ci-unit-tests-docker.yml Dismissed Show dismissed Hide dismissed

fix: nextest timeouts

59fd617

conradoplg reviewed Aug 6, 2025

View reviewed changes

zebrad/tests/acceptance.rs Outdated Show resolved Hide resolved

zebrad/tests/acceptance.rs Show resolved Hide resolved

zebrad/tests/acceptance.rs Show resolved Hide resolved

zebrad/tests/acceptance.rs Show resolved Hide resolved

gustavovalverde and others added 5 commits August 6, 2025 15:02

Apply suggestions from code review

a364067

Co-authored-by: Conrado Gouvea <conrado@zfnd.org>

fix: linting errors

c90c674

chore: update documentation and add missing profiles

d06e0f1

chore: more documentation fixes

0e84e1e

Merge branch 'main' into imp-caching

7463aa4

conradoplg approved these changes Aug 11, 2025

View reviewed changes

gustavovalverde mentioned this pull request Aug 23, 2025

feat(config)!: migrate zebrad to use a layered configuration #9768

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ci): implement nextest and optimize Docker test builds#9435

feat(ci): implement nextest and optimize Docker test builds#9435
gustavovalverde merged 46 commits intomainfrom
imp-caching

gustavovalverde commented Apr 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

conradoplg left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

conradoplg left a comment

Uh oh!

conradoplg commented Aug 11, 2025

Uh oh!

mergify bot commented Aug 11, 2025

Uh oh!

gustavovalverde commented Aug 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

gustavovalverde commented Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Solution

1. Nextest Integration with Centralized Configuration

2. Docker Build Optimization

3. CI/CD Workflow Modernization

4. Entrypoint Simplification (Phase 1)

5. Test Execution Improvements

Performance Impact

Migration Path

Testing

Uh oh!

Uh oh!

conradoplg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

conradoplg left a comment

Choose a reason for hiding this comment

Uh oh!

conradoplg commented Aug 11, 2025

Uh oh!

mergify bot commented Aug 11, 2025

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

Uh oh!

gustavovalverde commented Aug 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gustavovalverde commented Apr 17, 2025 •

edited

Loading