Truncated downloads from Cloudflare resulting in checksum mismatches

[richardlau]

@bnb reported that Electron were seeing downloads from the headers server not matching the checksums in the SHA256SUMS.txt file:

curl --compressed https://nodejs.org/download/release/v14.17.5/win-x86/node.lib |sha256sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  845k    0  845k    0     0  1242k      0 --:--:-- --:--:-- --:--:-- 1250k
49a6c038e4e53657b1b7a173c601e8ba1585d0f6dbfa88ac789dba107d596812  -

https://nodejs.org/download/release/v14.17.5/SHASUMS256.txt

1078be47b9315c81aa3bb989c4bba8ee23e0da9e4854a44006decf45d578833e  win-x86/node.lib

We had something similar reported four days ago with the node.lib file from a different release: nodejs/node#46594

In both cases the files on our two servers (www and unencrypted) still matched the checksum -- further investigation showed the file downloaded/cached from the nodejs.org URL was truncated. Purging CloudFlare's CDN cache[1] fixed both issues, although I'm concerned we've now seen this twice.

[1]: @nodejs/build-infra To purge the cache, log into www and run

/home/nodejs/queue-cdn-purge.sh nodejs <reason>

where <reason> is something like, e.g.

/home/nodejs/queue-cdn-purge.sh nodejs gh-build-3194

(The <reason> gets logged to /home/nodejs/cdn-purge.log.) Once queued the purge should happen within five minutes. The queue script will write /tmp/cdnpurge.nodejs and this file will be removed when the purge happens.

[richardlau]

CDN purge has fixed the reported file:

curl --compressed https://nodejs.org/download/release/v14.17.5/win-x86/node.lib |sha256sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1359k    0 1359k    0     0  2127k      0 --:--:-- --:--:-- --:--:-- 2127k
1078be47b9315c81aa3bb989c4bba8ee23e0da9e4854a44006decf45d578833e  -

Note that the lib file is 1349k now when previously only 845k was received.

[mikeharder]

Should all node versions be fixed now, or does each impacted node version need to be fixed independently? We have been seeing checksum mismatches in Node 14.21.2, both win-x64/node.lib and win-x86/node.lib.

[richardlau]

The CDN purge flushes the entire CDN cache for nodejs.org, causing CloudFlare to go back to our servers when fetching nodejs.org URLs the next time.

[richardlau]

@mikeharder were those checksum mismatches before the CDN purge ~3 hours ago or since?

[mikeharder]

@richardlau: After the CDN purge. We saw this 45 minutes ago, and I am trying to repro again right now.

[richardlau]

ugh, okay I can see https://nodejs.org/download/release/v14.21.2/win-x86/node.lib isn't matching the checksum and is truncated 😞. (But FWIW for https://nodejs.org/download/release/v14.21.2/win-x64/node.lib does match.)

[mikeharder]

Earlier today, we were seeing consistent mismatches in all our Windows builds. Around the time of the CDN purge, we started seeing a mix of matches and mismatches across our builds. As of 45 minutes ago, it appears to be back to consistent mismatches on Windows. No mismatches on Linux or Mac.

[mikeharder]

ugh, okay I can see https://nodejs.org/download/release/v14.21.2/win-x86/node.lib isn't matching the checksum and is truncated 😞. (But FWIW for https://nodejs.org/download/release/v14.21.2/win-x64/node.lib does match.)

How many times are you trying each download? It might be non-deterministic, where the download sometimes matches and sometimes doesn't.

[richardlau]

I've tried half a dozen times now for https://nodejs.org/download/release/v14.21.2/win-x64/node.lib but it's quite possible the correct file is being cached somewhere between me and our servers (e.g. some part of the CloudFlare CDN).

I'm really puzzled by this as it was first reported last week and as far as I know we haven't changed anything on our end -- the files on our server are intact and correct and something seems to have gone awry in the CloudFlare caching layer 😞.

[richardlau]

I'll queue another CDN purge (will take effect in about five minutes or so after queuing). It's past midnight here so I'm about to shut my laptop down.

[mikeharder]

@richardlau: Thanks for looking into this. If there is any specific information from our side that might help, feel free to ask.

[mikeharder]

Our most recent build got matching checksums for both win-x64/node.lib and win-x86/node.lib (no failures).

[mikeharder]

We haven't seen any more failures over the last 4 hours, so from our side this appears to be fixed for now. I will update this issue if we see any new failures.