fix(CI): Cache compilation results by upbqdn · Pull Request #9328 · ZcashFoundation/zebra

upbqdn · 2025-03-14T17:22:20Z

Motivation

Close CI recompiles Zebra from scratch for each CI test #9331.

Depends-On: #9344

Solution

Cache downloaded crates between image preparations: https://github.com/ZcashFoundation/zebra/actions/runs/13837431819/job/38722773169?pr=9323#step:10:367 vs https://github.com/ZcashFoundation/zebra/actions/runs/13867297186/job/38808809286?pr=9328#step:10:1.
Cache compiled crates between image preparations: https://github.com/ZcashFoundation/zebra/actions/runs/13837431819/job/38722773169?pr=9323#step:10:871 vs https://github.com/ZcashFoundation/zebra/actions/runs/13867297186/job/38808809286?pr=9328#step:10:387.
Cache compiled crates between image preparation and tests: https://github.com/ZcashFoundation/zebra/actions/runs/13837431819/job/38722794636?pr=9323#step:14:146 vs https://github.com/ZcashFoundation/zebra/actions/runs/13867297283/job/38809190703?pr=9328#step:14:134. For this test, the runtime went down from 33 minutes to 13 minutes.
Unify compilation parameters for all CI tests so caching can kick in.

Clean-ups

Mount lwd's and Zebra's cached states to implicit locations instead of explicitly specifying the location in each test.

Tests

CI is noticeably faster.

PR Checklist

The PR name is suitable for the release notes.
The solution is tested.
The documentation is up to date.
The PR has a priority label.
If the PR shouldn't be in the release notes, it has the
C-exclude-from-changelog label.

This reverts commit 960d5ca.

upbqdn · 2025-03-15T00:04:21Z

This PR seems to make CI noticeably faster.

upbqdn · 2025-03-15T00:42:12Z

We could bring the CI time down to ~25 minutes if we moved these three tests to run only on main:

upbqdn · 2025-03-17T11:52:11Z

docker/Dockerfile

+    cp -a ${CARGO_TARGET_DIR} ${HOME}/tmp_target  && \
+    cp -a ${CARGO_HOME} ${HOME}/tmp_cargo_home
+
+RUN rm -rf ${CARGO_TARGET_DIR} && rm -rf ${CARGO_HOME} && \
+    mv ${HOME}/tmp_target ${CARGO_TARGET_DIR}  && \
+    mv ${HOME}/tmp_cargo_home ${CARGO_HOME}


This is a rather awful hack. I think there could a cleaner solution, but I coulnd't come up with anything better.

Why is this needed?

This might be the best context to answer that #6169 (comment)

mergify · 2025-04-01T08:22:03Z

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 ⛓️ Depends-On Requirements

Wonderful, this rule succeeded.

Requirement based on the presence of Depends-On in the body of the pull request

depends-on = ZcashFoundation/zebra#9344 [⛓️ refactor(docker): Enhance Zebra configuration options and entrypoint logic #9344]

gustavovalverde · 2025-04-17T08:02:27Z

@upbqdn can you rebase this PR so we can re-test it?

Or we can use:

feat(ci): implement nextest and optimize Docker test builds #9435

upbqdn · 2025-04-17T09:45:38Z

This PR is based on the one we reverted. I don't think it's worth rebasing. It'll be easier to start from scratch again.

upbqdn added 27 commits March 11, 2025 18:10

Use gosu only once

5ae46cc

Remove COLORBT_SHOW_HIDDEN

960d5ca

Simplify Dockerfile

cd13024

Remove check_directory_files from entrypoint

9453508

Remove check for ZEBRA_CONF_PATH in entrypoint

b925925

Simplify ownership setting for ZEBRA_CACHE_DIR

32702ea

Simplify ownership setting for LOG_FILE

61c8a77

Refactor Dockerfile & entrypoint

5d3c691

Merge branch 'main' into fix-ci

70fc86d

Refactor vars in Dockerfile

1eff792

fmt

64aff28

Use chown for ZEBRA_CONF_PATH

1099cbe

run_cargo_test -> run_test

e8fb511

Make run_test runnable with gosu

d337755

Cosmetics

5cd63e8

Don't pre-compile Zebra

26d06de

Revert: "Don't pre-compile Zebra"

ac07975

Fix the custom conf test

e686db6

Reintroduce CARGO_HOME in Dockerfile

10394c1

Pass FEATURES as env var to entrypoint

91379b0

Fix ARGs in Dockerfile

a4c04e3

Revert "Remove COLORBT_SHOW_HIDDEN"

ffd5e11

This reverts commit 960d5ca.

Specify cache state dir in CI

43ce25b

Specify lwd cache dir in CI

9289ff4

fmt

158674e

Mount Zebra & lwd state dirs to static locations

4ea68a1

Remove redundant vars

d0e3277

upbqdn added A-devops Area: Pipelines, CI/CD and Dockerfiles C-cleanup Category: This is a cleanup P-Low ❄️ labels Mar 14, 2025

upbqdn temporarily deployed to dev March 14, 2025 19:15 — with GitHub Actions Inactive

Try fixing compilation caching

3e0536e

upbqdn temporarily deployed to dev March 14, 2025 20:50 — with GitHub Actions Inactive

upbqdn had a problem deploying to dev March 14, 2025 20:50 — with GitHub Actions Error

upbqdn temporarily deployed to dev March 14, 2025 20:51 — with GitHub Actions Inactive

upbqdn added 8 commits March 14, 2025 22:00

Use cp -a instead of cp -r

0670079

Ignore checkpoints

6464a6c

Fix syntax

c3e0be1

rm target

f5babd8

remove target and cargo home

39ec1a7

rm rm target

90fb2e8

Reinstantiate checkpoints

5e4cea9

Reinstantiate zebra-checkpoints binary

470b877

upbqdn commented Mar 17, 2025

View reviewed changes

upbqdn mentioned this pull request Mar 17, 2025

fix(ci): Better permission and cache dirs handling in Docker #9323

Merged

5 tasks

gustavovalverde mentioned this pull request Apr 10, 2025

fix(docker): Correct cargo dependency caching in build stages #9410

Closed

5 tasks

gustavovalverde mentioned this pull request Apr 17, 2025

feat(ci): implement nextest and optimize Docker test builds #9435

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(CI): Cache compilation results#9328

fix(CI): Cache compilation results#9328
upbqdn wants to merge 40 commits intomainfrom
fix-deploy-gcp

upbqdn commented Mar 14, 2025 •

edited by gustavovalverde

Loading

Uh oh!

upbqdn commented Mar 15, 2025

Uh oh!

upbqdn commented Mar 15, 2025

Uh oh!

upbqdn Mar 17, 2025

Uh oh!

conradoplg Mar 17, 2025

Uh oh!

gustavovalverde Mar 18, 2025

Uh oh!

mergify bot commented Apr 1, 2025 •

edited

Loading

Uh oh!

gustavovalverde commented Apr 17, 2025 •

edited

Loading

Uh oh!

upbqdn commented Apr 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

upbqdn commented Mar 14, 2025 • edited by gustavovalverde Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Solution

Clean-ups

Tests

PR Checklist

Uh oh!

upbqdn commented Mar 15, 2025

Uh oh!

upbqdn commented Mar 15, 2025

Uh oh!

upbqdn Mar 17, 2025

Choose a reason for hiding this comment

Uh oh!

conradoplg Mar 17, 2025

Choose a reason for hiding this comment

Uh oh!

gustavovalverde Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Protections

🟢 ⛓️ Depends-On Requirements

Uh oh!

gustavovalverde commented Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

upbqdn commented Apr 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

upbqdn commented Mar 14, 2025 •

edited by gustavovalverde

Loading

mergify bot commented Apr 1, 2025 •

edited

Loading

gustavovalverde commented Apr 17, 2025 •

edited

Loading