Another attempt at docker caching in CI

The default "gha" seems to be spraying so much into the cache that
everything's just gettting evicted for taking up too much space. I also
don't know why it's spraying so much. Try out something else which is
more targeted and should at least have a useful cache key name.

Author: alexcrichton

.github/actions/docker-build-cached/action.yml

@@ -0,0 +1,51 @@
+name: 'Build docker image'
+description: 'Build docker image'
+
+inputs:
+  context:
+    description: 'Context argument'
+    required: true
+  file:
+    description: 'Docker file'
+    required: true
+  cache_key_prefix:
+    description: 'Cache key prefix'
+    required: true
+
+runs:
+  using: composite
+  steps:
+    # Docker-recommended setup.
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v4
+
+    # Disable things that docker has on-by-default from the above which we're
+    # not interested in.
+    - name: Disable some docker things that are on-by-default
+      shell: bash
+      run: |
+        echo DOCKER_BUILD_CHECKS_ANNOTATIONS=false >> $GITHUB_ENV
+        echo DOCKER_BUILD_SUMMARY=false >> $GITHUB_ENV
+        echo DOCKER_BUILD_RECORD_UPLOAD=false >> $GITHUB_ENV
+
+    # We'll be using this cache key and the "local" caching mode of docker. This
+    # allows pretty fine-grained views into what's being cache. Experimentation
+    # found that the default `gha` caching mode sprays quite a lot into the
+    # cache such that everything keeps getting evicted for being over the size
+    # limit, so this is the next-best attempt to do something a bit more
+    # targeted.
+    - name: Setup Github Actions cache
+      uses: actions/cache@v5
+      with:
+        path: ${{ runner.tool_cache }}/docker-buildx-cache
+        key: ${{ inputs.cache_key_prefix }}-${{ hashFiles(inputs.file) }}
+
+    - name: Build docker image
+      uses: docker/build-push-action@v7
+      with:
+        context: ${{ inputs.context }}
+        file: ${{ inputs.file }}
+        tags: build-image
+        cache-from: type=local,src=${{ runner.tool_cache }}/docker-buildx-cache
+        cache-to: type=local,dest=${{ runner.tool_cache }}/docker-buildx-cache
+        outputs: type=docker

.github/workflows/main.yml

@@ -1297,40 +1297,19 @@ jobs:
       with:
         submodules: true
 
-    # Attempt to speed up building of docker images for Linux-based binaries by
-    # using docker's recommended way of building/caching with github actions.
-    # Notably this caches layers to github actions which enables subsequent
-    # reuse in future runs. The goal here is to make this step faster while
-    # additionally reducing network flakiness since the github actions cache
-    # generally works better than upstream package servers, or at least in
-    # theory.
+    # Build the docker image here, before it's built below, to have
+    # finer-grained control over caching. The way this is set up is that the
+    # build in this step is picked up by the later step automatically, and if
+    # this one's buggy the next one will just take a bit longer.
     #
-    # The way that this is designed is to build the image with docker-related
-    # github actions here primarily. This populates the github actions cache
-    # but also the local docker daemon cache too. Subsequently when the actual
-    # build script runs it'll attempt to rebuild the image and it should get a
-    # bunch of cache hits from here. This way we can configure the caching here
-    # within github actions without impacting the functionality of the script
-    # itself and continue enabling it to run locally.
-    - name: Set up Docker Buildx
-      uses: docker/setup-buildx-action@v4
+    # The goal here is to hit the network less because apt-get servers tend
+    # to be relatively flaky...
+    - uses: ./.github/actions/docker-build-cached
       if: ${{ matrix.env.DOCKER_IMAGE }}
-    - name: Disable some docker things that are on-by-default
-      run: |
-        echo DOCKER_BUILD_CHECKS_ANNOTATIONS=false >> $GITHUB_ENV
-        echo DOCKER_BUILD_SUMMARY=false >> $GITHUB_ENV
-        echo DOCKER_BUILD_RECORD_UPLOAD=false >> $GITHUB_ENV
-      if: ${{ matrix.env.DOCKER_IMAGE }}
-    - name: Build docker image
-      if: ${{ matrix.env.DOCKER_IMAGE }}
-      uses: docker/build-push-action@v7
       with:
         context: ci/docker
         file: ${{ matrix.env.DOCKER_IMAGE }}
-        tags: build-image
-        cache-from: type=gha
-        cache-to: type=gha,mode=max
-        outputs: type=docker
+        cache_key_prefix: docker-${{ matrix.target }}
 
     - uses: ./.github/actions/install-ninja
       if: '${{ !matrix.env.DOCKER_IMAGE }}'