Yarn PnP Linux Faliure on Install

[ham-zeal]

intro

hello @evanw , thanks for all the killer stuff. there's a new issue i've found that relates only to yarn PnP and only on linux. This Repo will allow you to reproduce, but to save time here is the issue:

it broke

how to see the issue?

run:

yarn --inline-builds

on any linux os.

how to see if not on linux

run:

yarn fail:docker

what do you see?

 > [5/5] RUN yarn --inline-builds:
#9 0.689 ➤ YN0000: ┌ Resolution step
#9 0.698 ➤ YN0000: └ Completed
#9 0.699 ➤ YN0000: ┌ Fetch step
#9 0.841 ➤ YN0000: └ Completed
#9 0.845 ➤ YN0000: ┌ Link step
#9 0.879 ➤ YN0008: │ esbuild@npm:0.13.3 must be rebuilt because its dependency tree changed
#9 1.192 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR /usr/app/.yarn/unplugged/esbuild-npm-0.13.3-0c72846e03/node_modules/esbuild/lib/yarn-pnp-esbuild: 1: /usr/app/.yarn/unplugged/esbuild-npm-0.13.3-0c72846e03/node_modules/esbuild/lib/yarn-pnp-esbuild: Syntax error: "&" unexpected
#9 1.192 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR child_process.js:830
#9 1.192 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     throw err;
#9 1.192 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     ^
#9 1.193 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR
#9 1.193 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR Error: Command failed: /usr/app/.yarn/unplugged/esbuild-npm-0.13.3-0c72846e03/node_modules/esbuild/lib/yarn-pnp-esbuild --version
#9 1.193 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at checkExecSyncError (child_process.js:790:11)
#9 1.193 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Object.execFileSync (child_process.js:827:15)
#9 1.194 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Object.<anonymous> (/usr/app/.yarn/unplugged/esbuild-npm-0.13.3-0c72846e03/node_modules/esbuild/bin/esbuild:99:26)
#9 1.194 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Module._compile (internal/modules/cjs/loader.js:1085:14)
#9 1.194 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
#9 1.194 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Module.load (internal/modules/cjs/loader.js:950:32)
#9 1.194 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Function.external_module_.Module._load (/usr/app/.pnp.cjs:9590:14)
#9 1.194 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
#9 1.194 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at internal/main/run_main_module.js:17:47 {
#9 1.194 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   status: 2,
#9 1.194 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   signal: null,
#9 1.195 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   output: [ null, null, null ],
#9 1.195 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   pid: 50,
#9 1.195 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   stdout: null,
#9 1.195 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   stderr: null
#9 1.195 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR }
#9 1.201 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR child_process.js:830
#9 1.201 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     throw err;
#9 1.202 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     ^
#9 1.202 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR
#9 1.202 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR Error: Command failed: node /usr/app/.yarn/unplugged/esbuild-npm-0.13.3-0c72846e03/node_modules/esbuild/bin/esbuild --version
#9 1.202 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR /usr/app/.yarn/unplugged/esbuild-npm-0.13.3-0c72846e03/node_modules/esbuild/lib/yarn-pnp-esbuild: 1: /usr/app/.yarn/unplugged/esbuild-npm-0.13.3-0c72846e03/node_modules/esbuild/lib/yarn-pnp-esbuild: Syntax error: "&" unexpected
#9 1.202 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR child_process.js:830
#9 1.202 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     throw err;
#9 1.202 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     ^
#9 1.202 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR
#9 1.202 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR Error: Command failed: /usr/app/.yarn/unplugged/esbuild-npm-0.13.3-0c72846e03/node_modules/esbuild/lib/yarn-pnp-esbuild --version
#9 1.202 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at checkExecSyncError (child_process.js:790:11)
#9 1.202 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Object.execFileSync (child_process.js:827:15)
#9 1.203 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Object.<anonymous> (/usr/app/.yarn/unplugged/esbuild-npm-0.13.3-0c72846e03/node_modules/esbuild/bin/esbuild:99:26)
#9 1.203 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Module._compile (internal/modules/cjs/loader.js:1085:14)
#9 1.203 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
#9 1.203 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Module.load (internal/modules/cjs/loader.js:950:32)
#9 1.203 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Function.external_module_.Module._load (/usr/app/.pnp.cjs:9590:14)
#9 1.203 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
#9 1.203 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at internal/main/run_main_module.js:17:47 {
#9 1.203 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   status: 2,
#9 1.203 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   signal: null,
#9 1.203 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   output: [ null, null, null ],
#9 1.203 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   pid: 50,
#9 1.204 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   stdout: null,
#9 1.204 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   stderr: null
#9 1.204 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR }
#9 1.204 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR
#9 1.204 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at checkExecSyncError (child_process.js:790:11)
#9 1.204 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Object.execFileSync (child_process.js:827:15)
#9 1.204 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at validateBinaryVersion (/usr/app/.yarn/unplugged/esbuild-npm-0.13.3-0c72846e03/node_modules/esbuild/install.js:64:32)
#9 1.204 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Object.<anonymous> (/usr/app/.yarn/unplugged/esbuild-npm-0.13.3-0c72846e03/node_modules/esbuild/install.js:98:3)
#9 1.204 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Module._compile (internal/modules/cjs/loader.js:1085:14)
#9 1.204 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
#9 1.204 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Module.load (internal/modules/cjs/loader.js:950:32)
#9 1.204 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Function.external_module_.Module._load (/usr/app/.pnp.cjs:9590:14)
#9 1.204 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
#9 1.204 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     at internal/main/run_main_module.js:17:47 {
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   status: 1,
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   signal: null,
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   output: [
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     null,
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     Buffer(0) [Uint8Array] [],
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     Buffer(1144) [Uint8Array] [
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR        47, 117, 115, 114,  47,  97, 112, 112,  47,  46, 121,  97,
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR       114, 110,  47, 117, 110, 112, 108, 117, 103, 103, 101, 100,
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR        47, 101, 115,  98, 117, 105, 108, 100,  45, 110, 112, 109,
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR        45,  48,  46,  49,  51,  46,  51,  45,  48,  99,  55,  50,
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR        56,  52,  54, 101,  48,  51,  47, 110, 111, 100, 101,  95,
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR       109, 111, 100, 117, 108, 101, 115,  47, 101, 115,  98, 117,
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR       105, 108, 100,  47, 108, 105,  98,  47, 121,  97, 114, 110,
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR        45, 112, 110, 112,  45, 101, 115,  98, 117, 105, 108, 100,
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR        58,  32,  49,  58,
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR       ... 1044 more items
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     ]
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   ],
#9 1.205 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   pid: 43,
#9 1.206 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   stdout: Buffer(0) [Uint8Array] [],
#9 1.206 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   stderr: Buffer(1144) [Uint8Array] [
#9 1.206 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR      47, 117, 115, 114,  47,  97, 112, 112,  47,  46, 121,  97,
#9 1.206 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     114, 110,  47, 117, 110, 112, 108, 117, 103, 103, 101, 100,
#9 1.206 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR      47, 101, 115,  98, 117, 105, 108, 100,  45, 110, 112, 109,
#9 1.206 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR      45,  48,  46,  49,  51,  46,  51,  45,  48,  99,  55,  50,
#9 1.206 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR      56,  52,  54, 101,  48,  51,  47, 110, 111, 100, 101,  95,
#9 1.206 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     109, 111, 100, 117, 108, 101, 115,  47, 101, 115,  98, 117,
#9 1.206 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     105, 108, 100,  47, 108, 105,  98,  47, 121,  97, 114, 110,
#9 1.206 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR      45, 112, 110, 112,  45, 101, 115,  98, 117, 105, 108, 100,
#9 1.206 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR      58,  32,  49,  58,
#9 1.206 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR     ... 1044 more items
#9 1.206 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR   ]
#9 1.206 ➤ YN0000: │ esbuild@npm:0.13.3 STDERR }
#9 1.210 ➤ YN0009: │ esbuild@npm:0.13.3 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-3033559e/build.log)
#9 1.210 ➤ YN0000: └ Completed in 0s 365ms
#9 1.215 ➤ YN0000: Failed with errors in 0s 527ms
------
executor failed running [/bin/sh -c yarn --inline-builds]: exit code: 1

does it work without pnp?

yes.

does it work on macos and windows?

yes, it's only on linux and only for PnP.

[evanw]

I tried using https://gitpod.io/#https://github.com/ham-zeal/esbuilb-yarn-pnp-linux-build-issue to run this on Linux, but I can't reproduce the problem:

gitpod /workspace/esbuilb-yarn-pnp-linux-build-issue $ node -p process.platform
linux

gitpod /workspace/esbuilb-yarn-pnp-linux-build-issue $ yarn --version
3.0.2

gitpod /workspace/esbuilb-yarn-pnp-linux-build-issue $ yarn --inline-builds
➤ YN0065: Yarn will periodically gather anonymous telemetry: https://yarnpkg.com/advanced/telemetry
➤ YN0065: Run yarn config set --home enableTelemetry 0 to disable

➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0000: └ Completed
➤ YN0000: ┌ Link step
➤ YN0007: │ esbuild@npm:0.13.3 must be built because it never has been before or the last one failed
➤ YN0000: └ Completed in 0s 400ms
➤ YN0000: Done in 0s 588ms

gitpod /workspace/esbuilb-yarn-pnp-linux-build-issue $ yarn fail:docker
Sending build context to Docker daemon  106.4MB
Step 1/5 : FROM node:14
14: Pulling from library/node
af4c2580c6c3: Pull complete 
4ea40d27a2cf: Pull complete 
523d612e9cd2: Pull complete 
8fee6a1847b0: Pull complete 
60cce3519052: Pull complete 
4bae1011637c: Pull complete 
ecf48b6c1f43: Pull complete 
856f69196742: Pull complete 
e86a512b6f8c: Pull complete 
Digest: sha256:167d0a4af6b4e0d0769086b871a36c25faed75b72705144cabbda70466cc0d8e
Status: Downloaded newer image for node:14
 ---> 8cf18d365a37
Step 2/5 : RUN mkdir -p /usr/app
 ---> Running in cfeff63e4079
Removing intermediate container cfeff63e4079
 ---> fdf535488e17
Step 3/5 : WORKDIR /usr/app
 ---> Running in 00364855109f
Removing intermediate container 00364855109f
 ---> 132a3fa018cf
Step 4/5 : COPY  . .
 ---> 13cab6bde93b
Step 5/5 : RUN yarn --inline-builds
 ---> Running in 1af598ada843
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0000: └ Completed
➤ YN0000: ┌ Link step
➤ YN0008: │ esbuild@npm:0.13.3 must be rebuilt because its dependency tree changed
➤ YN0000: └ Completed in 0s 348ms
➤ YN0000: Done in 0s 524ms
Removing intermediate container 1af598ada843
 ---> 425f83d7e3dd
Successfully built 425f83d7e3dd
Successfully tagged test:latest

Everything appears to work for me.

[evanw]

I did finally manage to reproduce the problem from scratch, but only by copying files that were already installed on macOS and then running them on Linux. In that case, Yarn's installation process breaks. This documentation seems relevant: https://yarnpkg.com/cli/rebuild/. Specifically this part:

Note that while Yarn forgets the compilation, the previous artifacts aren't erased from the filesystem and may affect the next builds (in good or bad). To avoid this, you may remove the .yarn/unplugged folder, or any other relevant location where packages might have been stored (Yarn may offer a way to do that automatically in the future).

So it looks like yarn's package management isn't idempotent. If you copy a directory tree containing a .yarn/unplugged folder from one OS to another, you need to delete that directory before running yarn again or the package directory may be corrupted. This appears to be by design. So the solution is to run some form of rm -fr .yarn/unplugged before running yarn. I've tried this solution out myself and fixes your issue. You have to delete the .yarn/unplugged folder after copying the files from the non-Linux OS to Linux but before running yarn.

[ham-zeal]

you are absolutely right my dude, points to you for the thoroughness. i've pushed an update to the repo to prove your point, it doesn't fail if we manually clear out unplugged as you've indicated. i'm happy to close as upstream and will open in yarn repo.

[evanw]

Can you try the version 0.13.4 of esbuild that I just released? I tried to make the install script more resilient in this situation even though it's not designed to be used like this. Now my Yarn PnP workaround should use separate paths for the executables for each platform so in theory multiple platforms can coexist simultaneously. So if you attempt to do this, I think it should actually work now. The trade-off is you'll end up wasting some file system space.

[ham-zeal]

you're dead right again my guy. this branch totally works even in the failure mode.

to be fair if you're using yarn PnP and complaining about disk space you've missed the whole point lol

[evanw]

Thanks for confirming! Closing this as fixed then.