Attempt to cache docker images in CI

Linux package servers have been particularly flaky today so I've
re-investigated how to maybe cache everything. The idea here is that we
use the "officially recommended" way of caching builds of docker images
on github actions, notably using the github actions cache. The way this
is configured is that some extra steps happen on github which populate
the github actions cache and local docker daemon such that when the
actual build happens, totally independently of these github actions
steps, it just so happens to get cache hits. This is done to ensure that
if we make a mistake here it doesn't result in stale builds, just slower
builds.

The overall hope here is that by using the github actions cache for
docker images we can hit package installation less than we currently do.
The hope is that all the images fit within the github actions cache. I
think they do but it's pretty noisy and each build generates new entries
so I'm not entirely sure what's happening. Local testing shows that
reruns do indeed proceed faster and don't hit `apt-get` for example,
though.

This doesn't entirely insulate us from issues with `apt-get` because
runs will still use `apt-get` externally from docker, such as just
installing a few extra packages. I don't know how to make those more
robust, but hopefully we can at least reduce some flakiness by caching
some things.

Author: alexcrichton

.github/workflows/main.yml

@@ -1297,7 +1297,44 @@ jobs:
       with:
         submodules: true
 
+    # Attempt to speed up building of docker images for Linux-based binaries by
+    # using docker's recommended way of building/caching with github actions.
+    # Notably this caches layers to github actions which enables subsequent
+    # reuse in future runs. The goal here is to make this step faster while
+    # additionally reducing network flakiness since the github actions cache
+    # generally works better than upstream package servers, or at least in
+    # theory.
+    #
+    # The way that this is designed is to build the image with docker-related
+    # github actions here primarily. This populates the github actions cache
+    # but also the local docker daemon cache too. Subsequently when the actual
+    # build script runs it'll attempt to rebuild the image and it should get a
+    # bunch of cache hits from here. This way we can configure the caching here
+    # within github actions without impacting the functionality of the script
+    # itself and continue enabling it to run locally.
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v4
+      if: ${{ matrix.env.DOCKER_IMAGE }}
+    - name: Disable some docker things that are on-by-default
+      run: |
+        echo DOCKER_BUILD_CHECKS_ANNOTATIONS=false >> $GITHUB_ENV
+        echo DOCKER_BUILD_SUMMARY=false >> $GITHUB_ENV
+        echo DOCKER_BUILD_RECORD_UPLOAD=false >> $GITHUB_ENV
+      if: ${{ matrix.env.DOCKER_IMAGE }}
+    - name: Build docker image
+      if: ${{ matrix.env.DOCKER_IMAGE }}
+      uses: docker/build-push-action@v7
+      with:
+        context: ci/docker
+        file: ${{ matrix.env.DOCKER_IMAGE }}
+        tags: build-image
+        cache-from: type=gha
+        cache-to: type=gha,mode=max
+        outputs: type=docker
+
     - uses: ./.github/actions/install-ninja
+      if: '${{ !matrix.env.DOCKER_IMAGE }}'
+
     - uses: ./.github/actions/install-rust
       with:
         toolchain: ${{ matrix.rust }}

ci/build-release-artifacts.sh

@@ -20,7 +20,7 @@ wrapper=""
 # have precise glibc requirements for Linux platforms for example.
 if [ "$DOCKER_IMAGE" != "" ]; then
   if [ -f "$DOCKER_IMAGE" ]; then
-    docker build --tag build-image --file $DOCKER_IMAGE ci/docker
+    docker buildx build -o type=docker --tag build-image --file $DOCKER_IMAGE ci/docker
     DOCKER_IMAGE=build-image
   fi