RFC: Leverage --incremental

[yacinehmito]

This is a feature request.

Issue

Our team is working on a large codebase and we are currently having issues with the time our CI takes. We are looking at all the ways to make the build faster.
One of the bottlenecks is the time it takes to run the tests. We investigated and most of time taken to run unit tests is not to actually run the tests but to compile the TS files on-the-fly.

Expected behavior

The thing is that TS files have already been compiled as part of the build step. We would love for ts-jest to leverage this and rely on them.

One idea is to have ts-jest write the compiled files on disk and leverage the incremental option. As ts-jest attempts to recompile the code, tsc will see that most of the work was already done.

We understand this is a large undertaking and I am ready to help! From a quick glance I already identified ts-jest performs additional transformation to hoist jest.mock() and that could be a challenge.

Alternatives

We found none of the speed-up alternatives to be satisfying. They are:

• Compile the test files ahead of time. It works great but we lose snapshots and it's a poor developer experience. Not a good fit.
• Compile with babel instead of ts-jest. It's still slow, it doesn't support all of TypeScript features, the TDD experience is not as good and the code that is tested in not the one that actually runs in production. Not a good fit.

We would largely prefer to stick with ts-jest but performance improvements are really necessary for us. Again, I'm ready to work on it.

What do you think about this feature, and would you support a PR introducing it?

[ahnpnl]

Hi,

We are happy to get community's help in improving ts-jest. Right now we are lack of capacity to support all needs from the community. Therefore feel free to help us by your PRs, we really appreciate.

[yacinehmito]

I attempted to work on this but currently ts-jest assumes that all compilation happens in memory and hands off caching to Jest. Leveraging --incremental goes against that.
It's an assumption that is relied on throughout the code so I couldn't refactor easily.

I'll try to do an incremental-only Jest transformer first and see how it behaves. After that we may consider introducing the feature to ts-jest but that will be a large undertaking.

[ahnpnl]

Have you checked ts-jest processing doc ? Does it help to bring up some ideas for your work ?

[yacinehmito]

I did after you told me about this. Thank you for the info.

It confirms the level of coupling. I did a PoC of a runner with support for --incremental and it turned out to be a bit slower than ts-jest.
With ts-jest, what we lose in speed by not using --incremental, we gained thanks to its aggressive caching.

The issue is that it's this very caching that makes us stop using ts-jest. We have quite a large TypeScript project (almost 300k lines of code) and ts-jest completely chokes on it. Even with a v8 set at 4GB of RAM it blows up. The culprit is the in-memory cache. We are relying on our PoC instead.

The PoC is less smart than ts-jest and a bit more hacky: it's a runner combined with a transformer.
The runner first compiles the project on disk with --incremental. Because it's a runner it happens only once; there are no concurrent compilation.
Then the transformer, instead of compiling the TS files, will just read the compiled JS file from disk, so it's quite fast.

In the end we ended up with something that is just slightly slower than ts-jest but simpler to reason about and with no pressure on the RAM.

---

The biggest challenge for the introduction of incremental in ts-jest is the TypeScript transformer to hoist calls of jest.mock. If we implement it naïvely, we might end up with running test files that were already on disk because they didn't build as part of a jest run, hence not having the hoisted jest.mock calls.

---

Takeaway: I don't think I'll be working on that feature for now. Sorry.

[ahnpnl]

Hmm interesting from your explanation. When ts-jest caching was written, there wasn’t incremental build at that point so I can understand why the choice is cached compilered stuffs on RAM.
I wonder if ts-jest somehow can reuse cache from tsBuildInfo instead of using current caching mechanism. Is it the way you tried with your POC ?

[yacinehmito]

Absolutely. It is however architecturally very different from ts-jest. I don't know how to transpose it to what ts-jest does.

---

Unrelated ramble about what I did:

I've been having a mitigated success scaling the PoC though. TypeScript can't emit a file with its dependencies. It means that I need to either emit all files before starting to run the tests, or build the dependency graph myself and emit the files one by one on demand. I'd rather not do that.

It's also more flaky than ts-jest because it analyses the dependencies statically; if there's a dynamic require it will be blind to it. That could be solved by having some sort of communication from the worker to the runner but it's not part of the API of jest-worker.

I also don't like the fact that I have to use a transformer. It means that the runtime first reads the TS file only to be told to read the JS file. I'd rather solve that at the resolver level but I didn't manage to implement a custom resolver for the runtime.

Despite all that, and without any sort of caching, on a large codebase it's just a bit slower than ts-jest in terms of speed (whether it's a hot or cold run). However in our case, because we compile first in our CI, we have a cold run with ts-jest but a hot run with the PoC.

I think there is tremendous opportunity lying here, but we'll have to wait a bit because, at the moment, Jest's and TypeScript's models seem to be fundamentally incompatible. The strategy of ts-jest, which is to transpile files on demand and aggressively cache everything, will stay the best one for almost all projects.

[ahnpnl]

Despite all that, and without any sort of caching, on a large codebase it's just a bit slower than ts-jest in terms of speed (whether it's a hot or cold run). However in our case, because we compile first in our CI, we have a cold run with ts-jest but a hot run with the PoC.

Is this you used --incremental for tsc and then use jest to run on compiled js files ? If that is the case, I can understand why jest and typescript's models are incompatible like your explanation below.

I think there is tremendous opportunity lying here, but we'll have to wait a bit because, at the moment, Jest's and TypeScript's models seem to be fundamentally incompatible. The strategy of ts-jest, which is to transpile files on demand and aggressively cache everything, will stay the best one for almost all projects.

I agreed

[yacinehmito]

Is this you used --incremental for tsc and then use jest to run on compiled js files ?

Almost. When you do that you lose the watcher and snapshots are improperly located. Here I have a runner that starts a TypeScript program, emits files with your hoist-jest TS transformer, and injects a custom transformer to the following tests. This transformer, when asked to transform the TS file now returns the JS file compiled on disk.

[ahnpnl]

FYI, perhaps later we can learn from Angular CLI regarding to this topic.

@yacinehmito I managed somehow to use create incremental program for ts-jest but I'm not sure if I did it correctly. If you have time, would you please help me to check my experiment ? You can find it on this branch. I made a folder called compiler and you can look into program.ts.

To run the experiment codes, ts-jest config needs to have

"experimental": true,
"compilerHost": true

and tsconfig needs to have incremental: true