Error: write after end when installing packages with npm 5.8.0 or higher

[jbcpollak]

Hi, I've been working with @zkat on an npm issue which appears to have a root cause in node-tar. The stack trace I am seeing is the following. We have been able to reproduce this with any version of npm > 5.8.0. (we have not tested with npm 5.7.x)

❯ rm -rf /tmp/empty-cache && rm -rf node_modules && npx npmc install --cache /tmp/empty-cache
npx: installed 763 in 28.723s
npm notice CANARY npmc is experimental software. If you find an issue, please file it in the main npm repository, and call out that you were using npmc.
/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/minipass/index.js:81
      throw new Error('write after end')
      ^

Error: write after end
    at MiniPass.write (/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/minipass/index.js:81:13)
    at ReadEntry.entry.on (/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/pacote/lib/extract-stream.js:30:21)
    at ReadEntry.emit (events.js:185:15)
    at ReadEntry.emit (/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/tar/node_modules/minipass/index.js:296:25)
    at ReadEntry.[maybeEmitEnd] (/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/tar/node_modules/minipass/index.js:249:12)
    at ReadEntry.end (/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/tar/node_modules/minipass/index.js:162:27)
    at Unpack.[consumeBody] (/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/tar/lib/parse.js:210:13)
    at Unpack.[consumeChunkSub] (/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/tar/lib/parse.js:391:40)
    at Unpack.[consumeChunk] (/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/tar/lib/parse.js:360:30)
    at Unzip.(anonymous function).on.chunk (/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/tar/lib/parse.js:291:59)
    at Unzip.emit (events.js:180:13)
    at Unzip.emit (/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/tar/node_modules/minipass/index.js:296:25)
    at Unzip.write (/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/tar/node_modules/minipass/index.js:99:17)
    at Unzip.write (/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/tar/node_modules/minizlib/index.js:284:29)
    at Unpack.write (/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/tar/lib/parse.js:307:19)
    at PassThrough.ondata (_stream_readable.js:651:20)
/Users/jpollak/.npm/_npx/99478/lib/node_modules/npmc/node_modules/minipass/index.js:81auth@0.9.7
      throw new Error('write after end')
      ^

[zkat]

@isaacs for context, this started happening for some people, sometimes, when working off network-sourced tarballs (rather than cached ones), and it's happening in the transform callback for pacote's extract-stream. The error was first reported when it was using PassThrough, but I changed it to use Minipass and stopped using the .end(....data...) format for last writes just in case, and shipped a canary for @jbcpollak to try, and it still reproed.

[jbcpollak]

One small clarification - for us this happens every time we do fresh install of our projects. We have a bootstrap script for new developers that checks out all our repos and runs npm i for them - it fails every time, for every developer with this error. I'm not sure if its always the same package - so I suppose it could be random, but we must have enough dependencies in our projects that it always fails.

[isaacs]

Fixed on latest pacote, in npm 6.1.

This issue is reproduced on macos

$ npm doctor

Check                               Value                        Recommendation
npm ping                            OK
npm -v                              v6.1.0
node -v                             v10.3.0
npm config get registry             https://registry.npmjs.org/
which git                           /usr/bin/git
Perms check on cached files         ok
Perms check on global node_modules  ok
Perms check on local node_modules   ok
Verify cache contents               verified 2 tarballs

installation log

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at writeAfterEnd (_stream_writable.js:243:12)
    at PassThrough.Writable.write (_stream_writable.js:291:5)
    at PassThrough.Writable.end (_stream_writable.js:579:10)
    at ReadEntry.entry.on (/usr/local/lib/node_modules/npm/node_modules/libcipm/node_modules/pacote/lib/extract-stream.js:19:41)
    at ReadEntry.emit (events.js:187:15)
    at ReadEntry.emit (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:301:25)
    at ReadEntry.[maybeEmitEnd] (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:252:12)
    at ReadEntry.end (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:165:27)
    at Unpack.[consumeBody] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:210:13)
    at Unpack.[consumeChunkSub] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:391:40)
    at Unpack.[consumeChunk] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:362:30)
    at Unzip.(anonymous function).on.chunk (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:291:59)
    at Unzip.emit (events.js:182:13)
    at Unzip.emit (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:301:25)
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:102:17)
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
Emitted 'error' event at:
    at writeAfterEnd (_stream_writable.js:245:10)
    at PassThrough.Writable.write (_stream_writable.js:291:5)
    [... lines matching original stack trace ...]
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
events.js:167
      throw er; // Unhandled 'error' event
      ^

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at writeAfterEnd (_stream_writable.js:243:12)
    at PassThrough.Writable.write (_stream_writable.js:291:5)
    at PassThrough.Writable.end (_stream_writable.js:579:10)
    at ReadEntry.entry.on (/usr/local/lib/node_modules/npm/node_modules/libcipm/node_modules/pacote/lib/extract-stream.js:19:41)
    at ReadEntry.emit (events.js:187:15)
    at ReadEntry.emit (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:301:25)
    at ReadEntry.[maybeEmitEnd] (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:252:12)
    at ReadEntry.end (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:165:27)
    at Unpack.[consumeBody] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:210:13)
    at Unpack.[consumeChunkSub] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:391:40)
    at Unpack.[consumeChunk] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:362:30)
    at Unzip.(anonymous function).on.chunk (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:291:59)
    at Unzip.emit (events.js:182:13)
    at Unzip.emit (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:301:25)
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:102:17)
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
Emitted 'error' event at:
    at writeAfterEnd (_stream_writable.js:245:10)
    at PassThrough.Writable.write (_stream_writable.js:291:5)
    [... lines matching original stack trace ...]
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
events.js:167
      throw er; // Unhandled 'error' event
      ^

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at writeAfterEnd (_stream_writable.js:243:12)
    at PassThrough.Writable.write (_stream_writable.js:291:5)
    at PassThrough.Writable.end (_stream_writable.js:579:10)
    at ReadEntry.entry.on (/usr/local/lib/node_modules/npm/node_modules/libcipm/node_modules/pacote/lib/extract-stream.js:19:41)
    at ReadEntry.emit (events.js:187:15)
    at ReadEntry.emit (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:301:25)
    at ReadEntry.[maybeEmitEnd] (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:252:12)
    at ReadEntry.end (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:165:27)
    at Unpack.[consumeBody] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:210:13)
    at Unpack.[consumeChunkSub] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:391:40)
    at Unpack.[consumeChunk] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:362:30)
    at Unzip.(anonymous function).on.chunk (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:291:59)
    at Unzip.emit (events.js:182:13)
    at Unzip.emit (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:301:25)
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:102:17)
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
Emitted 'error' event at:
    at writeAfterEnd (_stream_writable.js:245:10)
    at PassThrough.Writable.write (_stream_writable.js:291:5)
    [... lines matching original stack trace ...]
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
events.js:167
      throw er; // Unhandled 'error' event
      ^

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at writeAfterEnd (_stream_writable.js:243:12)
    at PassThrough.Writable.write (_stream_writable.js:291:5)
    at PassThrough.Writable.end (_stream_writable.js:579:10)
    at ReadEntry.entry.on (/usr/local/lib/node_modules/npm/node_modules/libcipm/node_modules/pacote/lib/extract-stream.js:19:41)
    at ReadEntry.emit (events.js:187:15)
    at ReadEntry.emit (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:301:25)
    at ReadEntry.[maybeEmitEnd] (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:252:12)
    at ReadEntry.end (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:165:27)
    at Unpack.[consumeBody] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:210:13)
    at Unpack.[consumeChunkSub] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:391:40)
    at Unpack.[consumeChunk] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:362:30)
    at Unzip.(anonymous function).on.chunk (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:291:59)
    at Unzip.emit (events.js:182:13)
    at Unzip.emit (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:301:25)
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:102:17)
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
Emitted 'error' event at:
    at writeAfterEnd (_stream_writable.js:245:10)
    at PassThrough.Writable.write (_stream_writable.js:291:5)
    [... lines matching original stack trace ...]
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
events.js:167
      throw er; // Unhandled 'error' event
      ^

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at writeAfterEnd (_stream_writable.js:243:12)
    at PassThrough.Writable.write (_stream_writable.js:291:5)
    at PassThrough.Writable.end (_stream_writable.js:579:10)
    at ReadEntry.entry.on (/usr/local/lib/node_modules/npm/node_modules/libcipm/node_modules/pacote/lib/extract-stream.js:19:41)
    at ReadEntry.emit (events.js:187:15)
    at ReadEntry.emit (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:301:25)
    at ReadEntry.[maybeEmitEnd] (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:252:12)
    at ReadEntry.end (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:165:27)
    at Unpack.[consumeBody] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:210:13)
    at Unpack.[consumeChunkSub] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:391:40)
    at Unpack.[consumeChunk] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:362:30)
    at Unzip.(anonymous function).on.chunk (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:291:59)
    at Unzip.emit (events.js:182:13)
    at Unzip.emit (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:301:25)
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:102:17)
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
Emitted 'error' event at:
    at writeAfterEnd (_stream_writable.js:245:10)
    at PassThrough.Writable.write (_stream_writable.js:291:5)
    [... lines matching original stack trace ...]
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
events.js:167
      throw er; // Unhandled 'error' event
      ^

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at writeAfterEnd (_stream_writable.js:243:12)
    at PassThrough.Writable.write (_stream_writable.js:291:5)
    at PassThrough.Writable.end (_stream_writable.js:579:10)
    at ReadEntry.entry.on (/usr/local/lib/node_modules/npm/node_modules/libcipm/node_modules/pacote/lib/extract-stream.js:19:41)
    at ReadEntry.emit (events.js:187:15)
    at ReadEntry.emit (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:301:25)
    at ReadEntry.[maybeEmitEnd] (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:252:12)
    at ReadEntry.end (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:165:27)
    at Unpack.[consumeBody] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:210:13)
    at Unpack.[consumeChunkSub] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:391:40)
    at Unpack.[consumeChunk] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:362:30)
    at Unzip.(anonymous function).on.chunk (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:291:59)
    at Unzip.emit (events.js:182:13)
    at Unzip.emit (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:301:25)
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:102:17)
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
Emitted 'error' event at:
    at writeAfterEnd (_stream_writable.js:245:10)
    at PassThrough.Writable.write (_stream_writable.js:291:5)
    [... lines matching original stack trace ...]
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
events.js:167
      throw er; // Unhandled 'error' event
      ^

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at writeAfterEnd (_stream_writable.js:243:12)
    at PassThrough.Writable.write (_stream_writable.js:291:5)
    at PassThrough.Writable.end (_stream_writable.js:579:10)
    at ReadEntry.entry.on (/usr/local/lib/node_modules/npm/node_modules/libcipm/node_modules/pacote/lib/extract-stream.js:19:41)
    at ReadEntry.emit (events.js:187:15)
    at ReadEntry.emit (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:301:25)
    at ReadEntry.[maybeEmitEnd] (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:252:12)
    at ReadEntry.end (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:165:27)
    at Unpack.[consumeBody] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:210:13)
    at Unpack.[consumeChunkSub] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:391:40)
    at Unpack.[consumeChunk] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:362:30)
    at Unzip.(anonymous function).on.chunk (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:291:59)
    at Unzip.emit (events.js:182:13)
    at Unzip.emit (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:301:25)
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minipass/index.js:102:17)
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)
Emitted 'error' event at:
    at writeAfterEnd (_stream_writable.js:245:10)
    at PassThrough.Writable.write (_stream_writable.js:291:5)
    [... lines matching original stack trace ...]
    at Unzip.write (/usr/local/lib/node_modules/npm/node_modules/tar/node_modules/minizlib/index.js:284:29)

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-06-12T10_11_27_506Z-debug.log
lerna ERR! npm ci exited 1 in 'root'
lerna ERR! npm ci exited 1 in 'root'

[VvorkCloudTechnologies]

no such file or directory, open '/home/g3006761319ishan/vvork-website/node_modules/.staging/core-js-aaa88aee/client/library.min.js.map'

i'm face the same issue