Enhancement: Major overhaul of installation

[lovell]

TL;DR - see #3750 (comment)

---

In 2015 sharp first started providing prebuilt libvips binaries and in 2018 started providing prebuilt binaries of itself.

The primary aim of this meta-improvement is to attempt to use only package manager mechanics at install time, without custom scripts, and without downloading binaries from hosts other than those controlled by a package manager.

(What appears below is a bit of a brain-dump and will evolve over time.)

What will this change allow/support?

Non-exhaustive list:

• Reproducible builds
• Offline installation
• Secure/sandbox environments where npm scripts cannot be run at install time
• Environments (countries, organisations) where access to GitHub is limited
• Vulnerability reporting via package manager for upstream dependencies

The approach taken by esbuild has now proven that this is possible with its use of optionalDependencies and platform-based cpu/os/libc filters.

A small but important piece of the puzzle fell into place with libc filtering via npm/npm-install-checks#54, which is part of npm v9.6.5 and therefore included with the latest Node.js 18 and 20. but sadly it's not quite available in the npm CLI yet - see npm/rfcs#438 (comment)

This should remove an entire class of install and runtime error, whilst no doubt introducing a new class of error, but a change is as good as a rest.

What needs to happen?

Register a suitable npm org for the namespace:

• @img - see https://www.npmjs.com/org/img
• Many thanks to the team at npm for transferring ownership.

Publish per-platform prebuilt binaries of libvips and its dependencies specifically for use by sharp:

• These will tale the form @img/sharp-libvips-platform-arch e.g. @img/sharp-libvips-linuxmusl-x64

Publish per-platform prebuilt binaries of sharp:

• These will take the form @sharpen/sharp-platform-arch e.g. @img/sharp-linuxmusl-x64
• Set rpath values for Linux and macOS that match the various package manager filesystem layouts.
  Example: if sharp.node is located at node_modules/@img/sharp-platform-arch/lib/sharp.node then its rpath needs to contain:
  • $$ORIGIN/../../sharp-libvips-platform-arch/lib
  • $$ORIGIN/../node_modules/@img/sharp-libvips-platform-arch/lib
  • Others?
• Provide 32-bit ARM binaries via QEMU that support lowest common denominator of ARMv6 arch as package managers do not differentiate on arm_version.

Support multi-platform within same installation tree:

• Consuming app/package can use dependencies to include a list of specific @img/sharp-platform-arch packages.
• Use yarn v3.1+ supportedArchitectures - see https://yarnpkg.com/configuration/yarnrc#supportedArchitectures
• npm has a (closed but not fully-implemented) RFC relating to this at [RRFC] Support --cpu and --os flag to specify platform specific install npm/rfcs#612 -

Support building from source:

• This will be opt-in, a breaking change. The assumption is that most people will not do this, so make it the exception rather than the rule.
• Compile sharp against prebuilt libvips:
  • Publish header files as common/generic @img/sharp-libvips-dev package and add to sharp's devDependencies
• Compile sharp against global libvips:
  • A consumer will need to add node-addon-api (and for yarn v3 node-gyp) to their package.json file.

Windows will be a bit different, natch:

• Need to ship sharp and libvips together as a single package to keep DLLs together = 8.9MB tarball
• Still need libvips-only packages and DLL copying logic etc. to allow (re)build of sharp from source e.g. @img/sharp-libvips-win32-x64.
• Need to do something different again for win32-arm64 (fewer than 30 downloads/month, so could drop then re-add later?)
• Currently no support for a globally-installed libvips DLLs, which will continue to be the case.

Support Wasm fallback:

• Will rely on the great work from WebAssembly build #3522
• Can we declare an optional package that does not support a list of otherwise-natively-supported os/cpu/libc combos, e.g. not linux-x64 and not darwin-x64 and not ...?
• What values do "sandbox" or "web container" platforms provide for process.platform and process.arch?
• Can we detect when Node.js is run with --no-addons at runtime? Yes, catch ERR_DLOPEN_DISABLED error code.
• Must display helpful error message for unsupported environments e.g. browsers and Wasm runtimes without threads/workers.

How is this going to be tested?

Need to be able to run sharp's unit tests against:

• sharp built from source against prebuilt libvips binaries.
• Prebuilt binaries of both sharp and libvips.
• Using non-Node.js runtimes too e.g. Deno, Bun.

What are the possible problems?

Impossible to please everyone all the time:

• Approach will fail when a package manager is configured to skip optional dependencies e.g. via npm's --no-optional flag.
• Filesize increase as Brotli-ball option for libvips binaries no longer an option = ~10% increased download size:
  • Can we use an npm postpack script to run advdef --recompress --shrink-insane over tarballs before uploading? No we cannot, as postpack is broken in slightly different ways in different versions of npm e.g. [BUG] npm publish with <tarball> parameter recreates tarball npm/cli#4533

Other breaking changes:

• Drop support for Node.js 14 and 16, minimum will be (semver) ^18.17.0 || ^20.3.0 || >=21.0.0
• Upgrade to latest Node-API version 9.
• Upgrade to libvips 8.15.0.
• Building from source becomes opt-in.
• Specify a minimum version of the various package managers based on support for libc limitation.
• yarn v3 PNP (Plug'n'Play) is unsupported, all the other nodeLinker settings should work. PNP support works.
• Drop support for EOL yarn v1:
  • It ignores os/cpu filters so people will complain about huge download/install e.g. Perf issue: npm/yarn download all platform builds! evanw/esbuild#2949
  • It generates errors for unknown engines so will require --ignore-engines flag.
  • If you're using yarn v1 there's never been a better time to upgrade.
• Remove sharp.vendor runtime API as this logic will now be "owned" by the package manager.
• Someone, somewhere, is going to realise for the first time that libvips is LGPL (sharp uses it under LGPLv3); insert "always has been" meme.