Cache versioned kubelet kubectl package binaries#8287
Cache versioned kubelet kubectl package binaries#8287awesomenix wants to merge 1 commit intomainfrom
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
awesomenix
requested review from
AbelHu,
Devinwong,
SriHarsha001,
calvin197,
cameronmeissner,
djsly,
ganeshkumarashok,
junjiezhang1997,
lilypan26,
mxj220,
pdamianov-dev,
phealy,
r2k1,
sulixu,
surajssd,
timmy-wright and
zachary-bailey
as code owners
PR Title Lint Failed ❌Current Title: Your PR title doesn't follow the expected format. Please update your PR title to follow one of these patterns: Conventional Commits Format:
Guidelines:
Examples:
Please update your PR title and the lint check will run again automatically. |
There was a problem hiding this comment.
Pull request overview
This PR optimizes Linux node provisioning by reusing kubelet/kubectl binaries already cached on the VHD (materialized as versioned files under /opt/bin) instead of reinstalling via the package manager during CSE.
Changes:
- VHD build: extract kubelet/kubectl binaries from cached
.deb/.rpmartifacts into/opt/bin/<tool>-<k8sVersion>. - CSE: add shared helpers to detect/move cached versioned kube binaries; update Ubuntu and Mariner package-based kubelet/kubectl install paths to prefer cache unless
SHOULD_ENFORCE_KUBE_PMC_INSTALL=true. - Tests: extend VHD content validation and add/extend ShellSpec coverage for the cache-first behavior.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
vhdbuilder/packer/install-dependencies.sh |
Adds VHD-build extraction of kubelet/kubectl binaries from cached package artifacts into versioned /opt/bin paths. |
vhdbuilder/packer/test/linux-vhd-content-test.sh |
Extends VHD content tests to validate versioned package-backed kubelet/kubectl binaries exist and match expected versions. |
parts/linux/cloud-init/artifacts/cse_install.sh |
Adds shared helpers to detect and move cached versioned kubelet/kubectl binaries; reuses them in the URL-based install flow. |
parts/linux/cloud-init/artifacts/ubuntu/cse_install_ubuntu.sh |
Updates Ubuntu package-based kubelet/kubectl install path to prefer cached versioned binaries. |
parts/linux/cloud-init/artifacts/mariner/cse_install_mariner.sh |
Updates Mariner/AzureLinux package-based kubelet/kubectl install path to prefer cached versioned binaries. |
spec/parts/linux/cloud-init/artifacts/cse_install_ubuntu_spec.sh |
Adds ShellSpec coverage for Ubuntu cache-first kubelet/kubectl install behavior. |
spec/parts/linux/cloud-init/artifacts/cse_install_mariner_spec.sh |
Adds ShellSpec coverage for Mariner cache-first kubelet/kubectl install behavior. |
| installKubeletKubectlFromURL() { | ||
| echo "installKubeletKubectlFromURL" | ||
| } | ||
|
|
||
| installPkgWithAptGet() { | ||
| echo "installPkgWithAptGet $*" | ||
| } | ||
|
|
||
| It 'uses cached kubernetes-binaries when both versioned binaries are already present' | ||
| hasCachedKubeBinariesForVersion() { | ||
| return 0 | ||
| } | ||
|
|
||
| When call installKubeletKubectlFromPkg "1.34.4" | ||
| The output should include "Found cached kubernetes-binaries for 1.34.4; installing from /opt/bin cache" | ||
| The output should include "installKubeletKubectlFromURL" | ||
| The output should not include "installPkgWithAptGet" | ||
| End |
There was a problem hiding this comment.
These tests stub installKubeletKubectlFromURL and assert it is called when cached binaries are present, but installKubeletKubectlFromPkg now calls moveCachedKubeBinariesForVersion directly. Without stubbing moveCachedKubeBinariesForVersion, the spec will execute the real mv/rm operations and likely fail. Update the spec to stub/assert moveCachedKubeBinariesForVersion instead of installKubeletKubectlFromURL.
| Describe 'installKubeletKubectlFromPkg' | ||
| installKubeletKubectlFromURL() { | ||
| echo "installKubeletKubectlFromURL" | ||
| } | ||
|
|
||
| installRPMPackageFromFile() { | ||
| echo "installRPMPackageFromFile $*" | ||
| } | ||
|
|
||
| It 'uses cached versioned kube binaries when available' | ||
| SHOULD_ENFORCE_KUBE_PMC_INSTALL="" | ||
| hasCachedKubeBinariesForVersion() { | ||
| return 0 | ||
| } | ||
|
|
||
| When call installKubeletKubectlFromPkg "1.34.4" | ||
| The output should include "Found cached kubernetes-binaries for 1.34.4; installing from /opt/bin cache" | ||
| The output should include "installKubeletKubectlFromURL" | ||
| The output should not include "installRPMPackageFromFile" | ||
| End |
There was a problem hiding this comment.
These tests stub installKubeletKubectlFromURL and assert it is called on cache hit, but installKubeletKubectlFromPkg now calls moveCachedKubeBinariesForVersion directly. Without stubbing moveCachedKubeBinariesForVersion, the spec will run the real mv/rm and fail. Update the spec to stub/assert moveCachedKubeBinariesForVersion instead.
| k8sVersion="${1}" | ||
| local k8sVersion="${1}" | ||
| if [ "${SHOULD_ENFORCE_KUBE_PMC_INSTALL}" != "true" ] && hasCachedKubeBinariesForVersion "${k8sVersion}"; then | ||
| echo "Found cached kubernetes-binaries for ${k8sVersion}; installing from /opt/bin cache" |
There was a problem hiding this comment.
The log line says "Found cached kubernetes-binaries..." but this path now also covers versioned kubelet/kubectl extracted from cached PMC packages (not just the kubernetes-binaries tarball). Consider changing the message to something source-agnostic (e.g., "Found cached kubelet/kubectl binaries for in /opt/bin").
| echo "Found cached kubernetes-binaries for ${k8sVersion}; installing from /opt/bin cache" | |
| echo "Found cached kubelet/kubectl binaries for ${k8sVersion} in /opt/bin" |
| installKubeletKubectlFromPkg() { | ||
| local desiredVersion="${1}" | ||
| if [ "${SHOULD_ENFORCE_KUBE_PMC_INSTALL}" != "true" ] && hasCachedKubeBinariesForVersion "${desiredVersion}"; then | ||
| echo "Found cached kubernetes-binaries for ${desiredVersion}; installing from /opt/bin cache" |
There was a problem hiding this comment.
The log line says "Found cached kubernetes-binaries..." but this path now also covers versioned kubelet/kubectl extracted from cached RPM/DEB packages (not just the kubernetes-binaries tarball). Consider changing the message to something source-agnostic (e.g., "Found cached kubelet/kubectl binaries for in /opt/bin").
| echo "Found cached kubernetes-binaries for ${desiredVersion}; installing from /opt/bin cache" | |
| echo "Found cached kubelet/kubectl binaries for ${desiredVersion} in /opt/bin" |
| return 1 | ||
| fi | ||
|
|
||
| rpm2cpio "${rpm_file}" | cpio -i --to-stdout "./usr/bin/${package_name}" "./usr/local/bin/${package_name}" | install -m0755 /dev/stdin "${binary_path}" | ||
| else |
There was a problem hiding this comment.
rpm2cpio | cpio -i --to-stdout is passed two possible binary paths (./usr/bin/${package_name} and ./usr/local/bin/${package_name}) and pipes stdout into install. With set -o pipefail enabled for this script, any non-zero from cpio will fail the VHD build, and if both paths ever exist in the RPM, cpio will emit both files to stdout sequentially, corrupting the cached binary. To make this robust, extract exactly one path (e.g., inspect the RPM file list first and then extract that single entry, or try /usr/bin/... and only fall back to /usr/local/bin/... if needed).
What this does
This change avoids unnecessary kubelet/kubectl package installation work during CSE when the corresponding binaries are already
available on the VHD.
Today, Ubuntu and Mariner/Azure Linux cache the kubelet/kubectl PMC packages on the VHD, but CSE still installs them via the package
manager/runtime package flow. That has a few downsides:
dpkg -itriggers packagepostinstbehavior forkubelet.serviceThis PR changes the flow so VHD build materializes versioned kubelet/kubectl binaries from the cached package artifacts into:
/opt/bin/kubelet-<k8sVersion>/opt/bin/kubectl-<k8sVersion>Then, during CSE, if the requested version is already present in
/opt/bin, we reuse the existing cache-first path and do the finalrename into place instead of reinstalling from the package.
Changes
VHD build
vhdbuilder/packer/install-dependencies.shso cachedkubelet/kubectlpackage artifacts also produce versioned binariesunder
/opt/bin/usr/bin/<tool>from cached.deb/usr/bin/<tool>from cached.rpmCSE
/opt/binSHOULD_ENFORCE_KUBE_PMC_INSTALL=trueTests
/opt/binWhy
This keeps kubelet/kubectl aligned with the existing
kubernetes-binariesflow:Expected benefits:
postinstside effects when the VHD already has the requested binaryNotes
SHOULD_ENFORCE_KUBE_PMC_INSTALL=truestill forces the package path for validation / test scenarios