Provide a way to load modules dynamically when npm starts without having to specify `NODE_OPTIONS`

[stefanobaghino]

As of npm 11, it's possible to use the node-options configuration key to specify the content of NODE_OPTIONS for lifecycle scripts (i.e. it wouldn't be picked up, for example, by npm test ... -- but it would be from npm pretest ...).

I'm wondering whether it would be feasible thinking about the idea of allowing users to opt-in into using this configuration key to set NODE_OPTIONS for npm commands as well.

I'm not hard-set on this solution, I'll share some context around why I'm making this proposal.

Context

As a software engineer at Gradle, I'm part of a team which is adding support for npm to our build observability product, Develocity.
We created a module that can be preloaded (NODE_OPTIONS="-r ...") and, when enabled, collects events from the run and sends them to the Develocity server.
We realized that setting NODE_OPTIONS can have ergonomics issues: setting it globally means that the module cannot be found for projects that don't have the package installed (and would be picked up by any Node.js process), and unfortunately using node-options from the npm config would not have an effect when running, for example npm test.

I'm open to discuss other solutions that would fit well with our use case of instrumenting npm.

[wraithgar]

I think the design as-is solves the problems it solves in the correct way. You either want NODE_OPTIONS set at a given context, to be set outside of npm, or you want it set for all the things npm calls.

I don't know where you got the idea that npm test wouldn't pick up the required module in this case.

~/D/s/scripts $ cat x.js
console.log('x is loaded')
~/D/s/scripts $ npm pkg get scripts.s
"semver 1.1.1"
~/D/s/scripts $ npm pkg get scripts.f
"npm run s"
~/D/s/scripts $ npm run s --node-options='-r ./x.js'

> scripts@1.0.0 s
> semver 1.1.1

x is loaded
1.1.1
~/D/s/scripts $ npm pkg get scripts.f
"npm run s"
~/D/s/scripts $ npm run f --node-options='-r ./x.js'

> scripts@1.0.0 f
> npm run s

x is loaded

> scripts@1.0.0 s
> semver 1.1.1

x is loaded
1.1.1

npm run always spawns a new node process, and it will have whatever --node-options you specify.

If you need the initial instance of npm itself to have node-options, you must set it in the environment beforehand. That's a classic bootstrapping problem there.

[wraithgar]

As of npm 11

This has been around since at least npm@6 fwiw. This is why we were comfortable w/ node documenting its use with npx. It already reasonably works for most folks using npm today.

[stefanobaghino]

If you need the initial instance of npm itself to have node-options, you must set it in the environment beforehand. That's a classic bootstrapping problem there.

This is exactly the problem we are having. In order to instrument npm itself, we need to ensure that NODE_OPTIONS is in the environment beforehand. However, asking users to always run NODE_OPTIONS="-r @gradle/develocity-agent/preload" npm test would make it quite cumbersome. Setting NODE_OPTIONS globally might interfere on projects where the module is not installed. We could ask to use a separate tool to manage environment variables, but ideally I would like this to be something that can be achieved without additional tools to avoid unnecessary complexity (and I can imagine there are scenarios where users are not necessarily free to install tooling locally).

I'm wondering whether a possible solution could be (and I can imagine this being an opt-in) to add a configuration key to enable this behavior (e.g. node-options-scope, defaults to lifecycle-scripts-only, could be set to full) and if enabled, npm forks itself with NODE_OPTIONS set from the configuration. We wouldn't still get access to the original process, but effectively this would allow to set NODE_OPTIONS from the config as if it was passed by setting the environment ahead of time.

I'm not hard-set neither on this solution or anything similar, I'm trying to figure out what tool we can use to reliably set preload modules when npm is run. I'm wondering if perhaps npm allows to list modules to be loaded in a way similar to how Node.js does with -r/--require. That would also solve our problem.

[wraithgar]

We wouldn't still get access to the original process.

This is already what's happening if you set node-options in npm config and run npm test.

What part of the original invocation of npm do you need, that you wouldn't need if npm were to re-invoke itself with node-options set? The delta between those two is not a lot of code.

[wraithgar]

Any solution that exists in npm itself is going to be adding complexity in places it by all rights shouldn't be. I understand the desire to have this managed invisibly to the user, but unfortunately setting the initial environment that runs when you invoke node is (and should remain) outside the scope of npm itself.

I know other tools and languages have scripts that put you into a shell mode where the environment is set as you need. This still seems like the right avenue. This is not complexity we need to add to npm.

[stefanobaghino]

What part of the original invocation of npm do you need, that you wouldn't need if npm were to re-invoke itself with node-options set? The delta between those two is not a lot of code.

Not too much indeed, although one thing that we need to track are command line arguments, which IIUC would be lost if NODE_OPTIONS is not available to the original npm invocation. But our problem is that we need to know that the current process is npm so that we can start instrumenting it. We then instrument any child process spawned by npm itself by propagating context through environment variables, but unless the process that first loads the module is npm, the instrumentation process doesn't start.

I know other tools and languages have scripts that put you into a shell mode where the environment is set as you need. This still seems like the right avenue. This is not complexity we need to add to npm.

Absolutely, I don't want to make npm a generic tool in this regard, but identifying a way in which a module can be loaded by npm itself when starting up. Do you think there's some other way to achieve this? Or alternatively, do you think there's some other way of instrumenting npm?

[stefanobaghino]

I gave this some additional thought and considered the following alternative: right now, for our use case, we use NODE_OPTIONS and check if the current process is npm. I considered the alternative of, instead, relying on node-options in the .npmrc and, from the lifecycle script, check if in the chain of parents of the current process, there's one that is npm. Unfortunately I realized that there's a fundamental flaw in this, as the lifecycle script might be some tool that is written in a different language, and would therefore be missed by our instrumentation.

Since setting NODE_OPTIONS gets stuck on a bootstrapping problem, and node-options in the .npmrc might cause us to miss specific edge cases (which I believe we'd like to make sure we cover), I'm wondering whether there's another way of having the effect of allowing a user configure a module to be loaded dynamically by npm itself before it starts.