Allow modules to be loaded from local directory

[tvanhens]

🚀 Feature Request

Affected Languages

• TypeScript or Javascript
• Python
• Java
• .NET (C#, F#, ...)

General Information

• JSII Version: n/a

• Platform: n/a

• I may be able to implement this feature request

• This feature might incur a breaking change

Description

While building a JSii wrapper for a Clojure CDK API, I've come to realize that it is beneficial to rely on npm & node_modules for JSii libraries. The bundling of the Java JSii runtime with the maven CDK modules opens the door to dependency conflicts that can be avoided by relying directly on node_modules. Additionally, maven modules are hard to load dynamically into a JVM which makes it hard to dynamically work with CDK constructs.

Right now the JSii protocol only allows for archived modules to be loaded (from the classpath in the Java case). If the JSii protocol was expanded to allow modules to be loaded locally from a node_modules folder it would make it easier to enable more dynamic workflows in REPL-friendly languages.

Proposed Solution

I can see two plausible solutions - both non-breaking:

1. Amend the load request to allow an optional boolean local argument. When this argument is supplied as true then modules are loaded from the local directory's node_modules folder. This would keep the surface area of the protocol largely unchanged and allows for a non-breaking change by falling back to the current behavior when the argument is not present.

2. Add a new require request that would allow a local module to be loaded. This increases the surface area of the API but prevents decreasing the complexity of the load API request. The commonality that would be shared between require and load could be factored into shared private functions to prevent drift between the two APIs.

I'm happy to put together a PR for either of these solutions (or an alternative solution) if this change is desirable.

[RomainMuller]

Hey,

I'd like to understand a bit more on your Clojure implementation here. Are you building on top of the Java libraries, or building a bindings generator for Clojure separately? (Note that while I know what Clojure is, I haven't quite used it for anything significant at this point).

Besides this, jsii may not rely on node forever. Having to have a compatible node runtime when developing in another language is a burden that we might eventually be able to do away with. This would imply switching to something maybe like WebAssembly in the future, where it won't be possible anymore to actually "load npm". We're actively trying to avoid customer-side dependency on npm precisely to avoid getting locked into it by features we cannot port out to a different execution engine.

[tvanhens]

Ah fair enough, hadn't thought about the underlying runtime changing that significantly. We are building on top of the jsii runtime for Java but not the generated classes. Instead, since Clojure can create namespaces and bindings at runtime, we generate the API wrappers in memory. Since these bindings don't have to be packaged and are generated from the underlying artifacts it is possible to pull new jsii artifacts in at runtime while using the REPL:

See https://github.com/StediInc/cdk-clj/blob/master/src/stedi/cdk/import.clj for an example

This uses the .jsii file contained in each jsii module to dynamically construct a Clojure-friendly API. I suppose another option for us would be to pull in the packages dynamically from GitHub releases, the drawback here is that we wouldn't be able to leverage the standardized coordinates of a package management system to retrieve non-cdk-core artifacts in a known way.

[RomainMuller]

@tvanhens - supposedly, all those packages (including 3P) are published to npmjs.com and can hence be retrieved using the standard npmjs.com API. The libraries are packaged as .tgz archives and the .jsii file is known to be located at package/.jsii (from memory, but wherever it's at in one such package is where you should look it up in any others - at least for now, and if that changes I reckon we'd put a field in the package.json field to actually indicate the desired location within the package).

[RomainMuller]

And by the way - if you're interested to contribute your Clojure handling directly into "mainland jsii", we'd welcome that ;)

In some future (talking months, not weeks here), I want to make the language support "pluggable" and this might give you more of the flexibility you require.

[github-actions]

This issue has not received any attention in 2 years. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.