-
Notifications
You must be signed in to change notification settings - Fork 30.9k
feat: use Rspack persistent cache by default #81399
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
66 commits
Select commit
Hold shift + click to select a range
cbcda2d
chore: update rspack to 1.4.2
SyMind b10df6d
chore: update rspack to 1.4.4
SyMind 97b405e
Merge branch 'canary' into rspack-1.4.2
SyMind d864ef3
update to 1.4.5
SyMind 913f571
chore: merge origin canary
SyMind 3a5986b
update snapshot
SyMind 84989f7
types-and-precompiled
SyMind a453c91
feat: open persistent cache by default in rspack
SyMind 00003e2
update snapshot
SyMind 1bf056f
chore: try rspack canary version
SyMind 73cc6e6
chore: merge origin canary
SyMind 7e36e8b
update canary
SyMind 5233590
merge origin canary
SyMind 0c9642d
update snapshot
SyMind 30aafa1
chore: merge origin canary
SyMind cabc286
rspack 1.4.8
SyMind e508d20
buildDependencies add babelConfigFile and jsConfigPath
SyMind 0912af3
fix: telemetry test case
SyMind 6089206
upgrade rspack
SyMind 91d3b18
chore: merge canary
SyMind 5fb33c2
feat: merge origin canary
SyMind adaf7c2
types-and-precompiled
SyMind d8503c2
chore: update rspack to 1.4.9
SyMind 0974c8f
chore: merge origin canary
SyMind e5e623d
types-and-precompiled
SyMind 2651036
init
SyMind 8098c80
HotReloaderRspack for Rspack
SyMind 61cad65
fix: should await super.start
SyMind bbf1b28
add lazy compilation middleware
SyMind 61aebda
chore: merge persisent cache
SyMind 4e07e55
builtEntriesCachePath
SyMind 4fc60d5
add hash for built entries cache
SyMind 1efdf43
merge
SyMind bfc2720
fix prettier
SyMind a688c81
fix: should not use createEntries
SyMind bd01aec
short hash
SyMind 9923d54
fix: typo
SyMind 6fbb92d
only cache entries that enabled cache
SyMind b8879c4
upgrade rspack
SyMind 62ed5cd
remove outdata entries
SyMind f769ce4
fix: file removed
SyMind b057a76
fix: test/integration/amphtml/test/index.test.js
SyMind 77f0dc6
chore: merge origin canary
SyMind 2a5a9d2
fix: pnpm lock
SyMind a34ffa9
fix: merge conflicts
SyMind edfa059
types-and-precompiled
SyMind c39c7e5
feat: using HotReloaderRspack
SyMind a098942
fix: cache buildDependencies
SyMind 64e191c
chore: rm test/integration/amphtml/test/index.test.js
SyMind 37fb20e
fix: cache for middleware
SyMind ba670a6
update rspack 1.6.5
SyMind b9f9000
update @next/rspack-core
SyMind 5239a8d
chore: merge origin canary
SyMind 5618d0a
chore: update snapshot
SyMind 714fa8d
chore: merge origin main
SyMind a657db5
lint
SyMind 21f31e6
udpate snapshot
SyMind dc4c455
Merge branch 'canary' into rspack-persistent-cache
SyMind 9c3bbcf
chore: merge origin canary
SyMind fa8f4ab
revert snapshot
SyMind 5861ba2
fix: mcp-server test case
SyMind 5549bc0
update snapshopt
SyMind ff21d1f
pnpm types-and-precompiled
SyMind 824879c
fix: buildFallbackError
SyMind 26eac3b
Merge branch 'canary' into rspack-persistent-cache
SyMind dc9af8a
add some comments
SyMind File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,243 @@ | ||
| import path from 'path' | ||
| import fs from 'fs/promises' | ||
| import { createHash } from 'crypto' | ||
| import HotReloaderWebpack from './hot-reloader-webpack' | ||
| import { BUILT, EntryTypes, getEntries } from './on-demand-entry-handler' | ||
| import type { __ApiPreviewProps } from '../api-utils' | ||
| import type { RouteDefinition } from '../route-definitions/route-definition' | ||
| import type { MultiCompiler } from 'webpack' | ||
| import { COMPILER_NAMES } from '../../shared/lib/constants' | ||
|
|
||
| /** | ||
| * Rspack Persistent Cache Strategy for Next.js Development | ||
| * | ||
| * Rspack's persistent caching differs from Webpack in how it manages module graphs. | ||
| * While Webpack incrementally updates modules, Rspack operates on complete module | ||
| * graph snapshots for cache restoration. | ||
| * | ||
| * Problem: | ||
| * - Next.js dev server starts with no page modules in the initial entry points | ||
| * - When Rspack restores from persistent cache, it finds no modules and purges | ||
| * the entire module graph | ||
| * - Later page requests find no cached module information, preventing cache reuse | ||
| * | ||
| * Solution: | ||
| * - Track successfully built page entries after each compilation | ||
| * - Restore these entries on dev server restart to maintain module graph continuity | ||
| * - This ensures previously compiled pages can leverage persistent cache for faster builds | ||
| */ | ||
| export default class HotReloaderRspack extends HotReloaderWebpack { | ||
| private builtEntriesCachePath?: string | ||
|
|
||
| private isClientCacheEnabled = false | ||
| private isServerCacheEnabled = false | ||
| private isEdgeServerCacheEnabled = false | ||
|
|
||
| public async afterCompile(multiCompiler: MultiCompiler): Promise<void> { | ||
| // Always initialize the fallback error watcher for Rspack. | ||
| // Rspack may restore/retain the previous build's error state, so without this | ||
| // a page that previously failed to build might not be rebuilt on the next request. | ||
| await super.buildFallbackError() | ||
|
|
||
| const rspackStartSpan = this.hotReloaderSpan.traceChild( | ||
| 'rspack-after-compile' | ||
| ) | ||
| await rspackStartSpan.traceAsyncFn(async () => { | ||
| const hash = createHash('sha1') | ||
| multiCompiler.compilers.forEach((compiler) => { | ||
| const cache = compiler.options.cache | ||
| if (typeof cache === 'object' && 'version' in cache && cache.version) { | ||
| hash.update(cache.version) | ||
| if (compiler.name === COMPILER_NAMES.client) { | ||
| this.isClientCacheEnabled = true | ||
| } else if (compiler.name === COMPILER_NAMES.server) { | ||
| this.isServerCacheEnabled = true | ||
| } else if (compiler.name === COMPILER_NAMES.edgeServer) { | ||
| this.isEdgeServerCacheEnabled = true | ||
| } | ||
| } else { | ||
| hash.update('-') | ||
| } | ||
| return undefined | ||
| }) | ||
| this.builtEntriesCachePath = path.join( | ||
| this.distDir, | ||
| 'cache', | ||
| 'rspack', | ||
| hash.digest('hex').substring(0, 16), | ||
| 'built-entries.json' | ||
| ) | ||
|
|
||
| const hasBuiltEntriesCache = await fs | ||
| .access(this.builtEntriesCachePath) | ||
| .then( | ||
| () => true, | ||
| () => false | ||
| ) | ||
| if (hasBuiltEntriesCache) { | ||
| try { | ||
| const builtEntries: ReturnType<typeof getEntries> = JSON.parse( | ||
| (await fs.readFile(this.builtEntriesCachePath, 'utf-8')) || '{}' | ||
| ) | ||
|
|
||
| await Promise.all( | ||
| Object.keys(builtEntries).map(async (entryKey) => { | ||
| const entryData = builtEntries[entryKey] | ||
|
|
||
| const isEntry = entryData.type === EntryTypes.ENTRY | ||
| const isChildEntry = entryData.type === EntryTypes.CHILD_ENTRY | ||
|
|
||
| // Check if the page was removed or disposed and remove it | ||
| if (isEntry) { | ||
| const pageExists = | ||
| !entryData.dispose && | ||
| (await fs.access(entryData.absolutePagePath).then( | ||
| () => true, | ||
| () => false | ||
| )) | ||
| if (!pageExists) { | ||
| delete builtEntries[entryKey] | ||
| return | ||
| } else if ( | ||
| !('hash' in builtEntries[entryKey]) || | ||
| builtEntries[entryKey].hash !== | ||
| (await calculateFileHash(entryData.absolutePagePath)) | ||
| ) { | ||
| delete builtEntries[entryKey] | ||
| return | ||
| } | ||
| } | ||
|
|
||
| // For child entries, if it has an entry file and it's gone, remove it | ||
| if (isChildEntry) { | ||
| if (entryData.absoluteEntryFilePath) { | ||
| const pageExists = | ||
| !entryData.dispose && | ||
| (await fs.access(entryData.absoluteEntryFilePath).then( | ||
| () => true, | ||
| () => false | ||
| )) | ||
| if (!pageExists) { | ||
| delete builtEntries[entryKey] | ||
| return | ||
| } else { | ||
| if ( | ||
| !('hash' in builtEntries[entryKey]) || | ||
| builtEntries[entryKey].hash !== | ||
| (await calculateFileHash( | ||
| entryData.absoluteEntryFilePath | ||
| )) | ||
| ) { | ||
| delete builtEntries[entryKey] | ||
| return | ||
| } | ||
| } | ||
| } | ||
| } | ||
| }) | ||
| ) | ||
| Object.assign(getEntries(multiCompiler.outputPath), builtEntries) | ||
| } catch (error) { | ||
| console.error('Rspack failed to read built entries cache: ', error) | ||
| } | ||
| } | ||
| }) | ||
| } | ||
|
|
||
| public async ensurePage({ | ||
| page, | ||
| clientOnly, | ||
| appPaths, | ||
| definition, | ||
| isApp, | ||
| url, | ||
| }: { | ||
| page: string | ||
| clientOnly: boolean | ||
| appPaths?: ReadonlyArray<string> | null | ||
| isApp?: boolean | ||
| definition?: RouteDefinition | ||
| url?: string | ||
| }): Promise<void> { | ||
| await super.ensurePage({ | ||
| page, | ||
| clientOnly, | ||
| appPaths, | ||
| definition, | ||
| isApp, | ||
| url, | ||
| }) | ||
| const entries = getEntries(this.multiCompiler!.outputPath) | ||
| const builtEntries: { [entryName: string]: any } = {} | ||
| await Promise.all( | ||
| Object.keys(entries).map(async (entryName) => { | ||
| const entry = entries[entryName] | ||
| if (entry.status !== BUILT) return | ||
| const result = | ||
| /^(client|server|edge-server)@(app|pages|root)@(.*)/g.exec(entryName) | ||
|
SyMind marked this conversation as resolved.
|
||
| const [, key /* pageType */, ,] = result! // this match should always happen | ||
| if (key === 'client' && !this.isClientCacheEnabled) return | ||
| if (key === 'server' && !this.isServerCacheEnabled) return | ||
| if (key === 'edge-server' && !this.isEdgeServerCacheEnabled) return | ||
|
|
||
| // TODO: Rspack does not store middleware entries in persistent cache, causing | ||
| // test/integration/middleware-src/test/index.test.ts to fail. This is a temporary | ||
| // workaround to skip middleware entry caching until Rspack properly supports it. | ||
| if (page === '/middleware') { | ||
| return | ||
| } | ||
|
|
||
| let hash: string | undefined | ||
| if (entry.type === EntryTypes.ENTRY) { | ||
| hash = await calculateFileHash(entry.absolutePagePath) | ||
| } else if (entry.absoluteEntryFilePath) { | ||
| hash = await calculateFileHash(entry.absoluteEntryFilePath) | ||
| } | ||
| if (!hash) { | ||
| return | ||
| } | ||
|
|
||
| builtEntries[entryName] = entry | ||
| builtEntries[entryName].hash = hash | ||
| }) | ||
| ) | ||
|
|
||
| const hasBuitEntriesCache = await fs | ||
| .access(this.builtEntriesCachePath!) | ||
| .then( | ||
| () => true, | ||
| () => false | ||
| ) | ||
| try { | ||
| if (!hasBuitEntriesCache) { | ||
| await fs.mkdir(path.dirname(this.builtEntriesCachePath!), { | ||
| recursive: true, | ||
| }) | ||
| } | ||
| await fs.writeFile( | ||
| this.builtEntriesCachePath!, | ||
| JSON.stringify(builtEntries, null, 2) | ||
| ) | ||
| } catch (error) { | ||
| console.error('Rspack failed to write built entries cache: ', error) | ||
| } | ||
| } | ||
| } | ||
|
|
||
| async function calculateFileHash( | ||
| filePath: string, | ||
| algorithm: string = 'sha256' | ||
| ): Promise<string | undefined> { | ||
| if ( | ||
| !(await fs.access(filePath).then( | ||
| () => true, | ||
| () => false | ||
| )) | ||
| ) { | ||
| return | ||
| } | ||
| const fileBuffer = await fs.readFile(filePath) | ||
| const hash = createHash(algorithm) | ||
| hash.update(fileBuffer) | ||
| return hash.digest('hex') | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.