Skip to content

Commit 7efd61a

Browse files
committed
Make files load fast - phase 1
1 parent 9a33de8 commit 7efd61a

6 files changed

Lines changed: 436 additions & 155 deletions

File tree

docs/adr/009-non-reactive-file-store.md

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -31,14 +31,14 @@ the full array assignment internally.
3131

3232
**Benchmark data (50k files):**
3333

34-
| Step | Time |
35-
| ---------------------- | --------- |
36-
| Rust list_directory | 308ms |
37-
| JSON serialize (Rust) | 18ms |
38-
| IPC transfer (17.4 MB) | ~4,100ms |
39-
| JSON.parse (JS) | 67ms |
40-
| Svelte reactivity | **9.5s** |
41-
| **Total** | **~14s** |
34+
| Step | Time |
35+
| ---------------------- | -------- |
36+
| Rust list_directory | 308ms |
37+
| JSON serialize (Rust) | 18ms |
38+
| IPC transfer (17.4 MB) | ~4,100ms |
39+
| JSON.parse (JS) | 67ms |
40+
| Svelte reactivity | **9.5s** |
41+
| **Total** | **~14s** |
4242

4343
The Svelte reactivity step alone takes 9.5 seconds for 50k files—this is the freeze the user experiences.
4444

@@ -70,8 +70,8 @@ Load files into Svelte state in small chunks (500 at a time) to spread out the r
7070

7171
**Option 1: Non-reactive FileDataStore + visible-range requests**
7272

73-
Create a plain JavaScript class that holds all file data outside of Svelte's reactivity system. Virtual scroll components
74-
request only the visible range of items to put into a small reactive array.
73+
Create a plain JavaScript class that holds all file data outside of Svelte's reactivity system. Virtual scroll
74+
components request only the visible range of items to put into a small reactive array.
7575

7676
**Architecture:**
7777

docs/notes/2025-12-29-make-files-load-super-fast-tasks.md

Lines changed: 57 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -4,38 +4,38 @@ Related: [2025-12-30-load-files-super-fast-spec.md](./2025-12-30-load-files-supe
44

55
---
66

7-
## Phase 1: FileDataStore (ADR-009)
7+
## Phase 1: FileDataStore (ADR-009)
88

99
Goal: Eliminate Svelte reactivity freeze by keeping file data outside reactive state.
1010

11-
- [ ] Create `FileDataStore` class (plain JS, not Svelte)
12-
- [ ] `files: FileEntry[]` — plain array storage
13-
- [ ] `totalCount: number` — for scrollbar sizing
14-
- [ ] `maxFilenameWidth: number` — for Brief mode horizontal scrollbar
15-
- [ ] `getRange(start, end): FileEntry[]` — for virtual scroll
16-
- [ ] `setFiles(entries: FileEntry[]): void` — bulk set
17-
- [ ] `appendFiles(entries: FileEntry[]): void` — for chunked loading
18-
- [ ] `clear(): void` — on navigation
19-
- [ ] `onUpdate` callback mechanism — notify components of changes
20-
21-
- [ ] Implement Brief mode width calculation
22-
- [ ] Use `canvas.measureText()` to measure filename widths
23-
- [ ] Calculate incrementally (first chunk immediately, rest in idle callback)
24-
25-
- [ ] Update `FilePane.svelte` to use FileDataStore
26-
- [ ] Create store instance per pane
27-
- [ ] Replace `allFilesRaw` with store
28-
- [ ] Request visible range on scroll
29-
- [ ] Update only `visibleItems` reactive state (~50-100 items)
30-
31-
- [ ] Update `BriefList.svelte` and `FullList.svelte`
32-
- [ ] Accept `totalCount` prop for scrollbar sizing
33-
- [ ] Accept `maxFilenameWidth` prop (Brief mode)
34-
- [ ] On scroll, call back to parent for new visible range
35-
36-
- [ ] Remove old `filesVersion` pattern
37-
- [ ] Delete `filesVersion` state variable
38-
- [ ] Delete `$derived.by()` that reads filesVersion
11+
- [x] Create `FileDataStore` class (plain JS, not Svelte)
12+
- [x] `files: FileEntry[]` — plain array storage
13+
- [x] `totalCount: number` — for scrollbar sizing
14+
- [x] `maxFilenameWidth: number` — for Brief mode horizontal scrollbar
15+
- [x] `getRange(start, end): FileEntry[]` — for virtual scroll
16+
- [x] `setFiles(entries: FileEntry[]): void` — bulk set
17+
- [x] `appendFiles(entries: FileEntry[]): void` — for chunked loading
18+
- [x] `clear(): void` — on navigation
19+
- [x] `onUpdate` callback mechanism — notify components of changes
20+
21+
- [x] Implement Brief mode width calculation
22+
- [x] Use `canvas.measureText()` to measure filename widths
23+
- [x] Calculate in `measureFilenameWidths()` function
24+
25+
- [x] Update `FilePane.svelte` to use FileDataStore
26+
- [x] Create store instance per pane
27+
- [x] Replace `allFilesRaw` with store
28+
- [x] Store provides `getAllFiltered()` for current implementation (full virtual scroll getRange coming in Phase 2)
29+
- [x] `storeVersion` reactive trigger for component updates
30+
31+
- [ ] Update `BriefList.svelte` and `FullList.svelte` (deferred to Phase 2)
32+
- [ ] Accept `totalCount` prop for scrollbar sizing
33+
- [ ] Accept `maxFilenameWidth` prop (Brief mode)
34+
- [ ] On scroll, call back to parent for new visible range
35+
36+
- [x] Remove old `filesVersion` pattern
37+
- [x] Delete `filesVersion` state variable (replaced with `storeVersion`)
38+
- [x] Delete direct `allFilesRaw` mutations
3939

4040
---
4141

@@ -44,27 +44,27 @@ Goal: Eliminate Svelte reactivity freeze by keeping file data outside reactive s
4444
Goal: Show files fast with core data, load extended metadata in background.
4545

4646
- [ ] Split `FileEntry` into core vs extended fields
47-
- [ ] Core: name, path, isDirectory, isSymlink, size, modifiedAt, createdAt, permissions, owner, group, iconId
48-
- [ ] Extended: addedAt, openedAt (macOS-specific)
49-
- [ ] Add `extendedMetadataLoaded: boolean` flag
47+
- [ ] Core: name, path, isDirectory, isSymlink, size, modifiedAt, createdAt, permissions, owner, group, iconId
48+
- [ ] Extended: addedAt, openedAt (macOS-specific)
49+
- [ ] Add `extendedMetadataLoaded: boolean` flag
5050

5151
- [ ] Update Rust `list_directory()` to support phased loading
52-
- [ ] New command: `list_directory_core()` — fast stat() only
53-
- [ ] New command: `list_directory_extended()` — macOS metadata
54-
- [ ] Add `// TODO: Apply sort criteria here` placeholder for future sorting
52+
- [ ] New command: `list_directory_core()` — fast stat() only
53+
- [ ] New command: `list_directory_extended()` — macOS metadata
54+
- [ ] Add `// TODO: Apply sort criteria here` placeholder for future sorting
5555

5656
- [ ] Update Rust session management
57-
- [ ] `list_directory_start` returns count + starts background loading
58-
- [ ] `list_directory_next_chunk` returns core data chunks
59-
- [ ] New: `list_directory_get_extended` returns extended metadata
57+
- [ ] `list_directory_start` returns count + starts background loading
58+
- [ ] `list_directory_next_chunk` returns core data chunks
59+
- [ ] New: `list_directory_get_extended` returns extended metadata
6060

6161
- [ ] Update `FileDataStore` for extended data
62-
- [ ] `mergeExtendedData(entries: PartialFileEntry[]): void` — merge by path
63-
- [ ] Track which items have extended data loaded
62+
- [ ] `mergeExtendedData(entries: PartialFileEntry[]): void` — merge by path
63+
- [ ] Track which items have extended data loaded
6464

6565
- [ ] Update UI to handle missing extended metadata
66-
- [ ] Show placeholder or omit fields if `extendedMetadataLoaded === false`
67-
- [ ] Update display when extended data arrives
66+
- [ ] Show placeholder or omit fields if `extendedMetadataLoaded === false`
67+
- [ ] Update display when extended data arrives
6868

6969
---
7070

@@ -73,12 +73,12 @@ Goal: Show files fast with core data, load extended metadata in background.
7373
Goal: Prepare for future sorting feature.
7474

7575
- [ ] Add sorting placeholders in Rust
76-
- [ ] `// TODO: Apply sort criteria here` before chunking
77-
- [ ] Document that first chunk should contain "best" files for current sort
76+
- [ ] `// TODO: Apply sort criteria here` before chunking
77+
- [ ] Document that first chunk should contain "best" files for current sort
7878

7979
- [ ] Add sorting placeholders in FileDataStore
80-
- [ ] `sortBy(criteria): void` method (stub)
81-
- [ ] Note: Re-sorting requires re-requesting from backend (sorted order affects which files are "first")
80+
- [ ] `sortBy(criteria): void` method (stub)
81+
- [ ] Note: Re-sorting requires re-requesting from backend (sorted order affects which files are "first")
8282

8383
---
8484

@@ -87,13 +87,13 @@ Goal: Prepare for future sorting feature.
8787
Goal: Stop wasting backend resources when user navigates away quickly.
8888

8989
- [ ] Add cancellation flag in Rust session
90-
- [ ] `is_cancelled: AtomicBool` in session struct
91-
- [ ] Check flag periodically in `list_directory()` loop
92-
- [ ] Exit early if cancelled
90+
- [ ] `is_cancelled: AtomicBool` in session struct
91+
- [ ] Check flag periodically in `list_directory()` loop
92+
- [ ] Exit early if cancelled
9393

9494
- [ ] Wire up cancellation from frontend
95-
- [ ] `list_directory_end_session` sets cancellation flag
96-
- [ ] Background thread checks flag and exits
95+
- [ ] `list_directory_end_session` sets cancellation flag
96+
- [ ] Background thread checks flag and exits
9797

9898
- [ ] Consider using `tokio` for proper async cancellation (complex, evaluate ROI)
9999

@@ -104,16 +104,16 @@ Goal: Stop wasting backend resources when user navigates away quickly.
104104
Goal: Optimize chunk sizes and timing.
105105

106106
- [ ] Benchmark different chunk sizes (1000, 2500, 5000, 10000)
107-
- [ ] Measure IPC overhead vs. perceived responsiveness
108-
- [ ] Document optimal size for different directory sizes
107+
- [ ] Measure IPC overhead vs. perceived responsiveness
108+
- [ ] Document optimal size for different directory sizes
109109

110110
- [ ] Consider dynamic chunk sizing
111-
- [ ] Smaller first chunk for faster initial display
112-
- [ ] Larger subsequent chunks for efficiency
111+
- [ ] Smaller first chunk for faster initial display
112+
- [ ] Larger subsequent chunks for efficiency
113113

114114
- [ ] Profile and optimize `canvas.measureText()` if needed
115-
- [ ] Batch measurements
116-
- [ ] Use `requestIdleCallback` for large directories
115+
- [ ] Batch measurements
116+
- [ ] Use `requestIdleCallback` for large directories
117117

118118
---
119119

docs/notes/2025-12-30-load-files-super-fast-spec.md

Lines changed: 37 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ Make the file explorer feel blazing fast when loading large directories (20k–1
77
1. Returning file count immediately (~16ms) to show progress
88
2. Sending "core" file data (name, size, dates, permissions) first for fast display
99
3. Loading extended metadata (lastOpenedDate, etc.) in parallel
10-
4. Using a non-reactive `FileDataStore` to avoid Svelte reactivity freezes (see [ADR-009](../adr/009-non-reactive-file-store.md))
10+
4. Using a non-reactive `FileDataStore` to avoid Svelte reactivity freezes (see
11+
[ADR-009](../adr/009-non-reactive-file-store.md))
1112

1213
Target: Match Commander One performance (~3s for 50k files to full load).
1314

@@ -19,16 +20,17 @@ See [2025-12-28-dir-load-bench-findings.md](./2025-12-28-dir-load-bench-findings
1920

2021
**Current bottlenecks:**
2122

22-
| Step | Time (50k files) |
23-
| ---------------------------- | ---------------- |
24-
| Rust list_directory | 308ms |
25-
| JSON serialize | 18ms |
26-
| IPC transfer (17.4 MB) | ~4,100ms |
27-
| JSON.parse | 67ms |
28-
| Svelte reactivity | ~9,500ms |
29-
| **Total** | **~14s** |
23+
| Step | Time (50k files) |
24+
| ---------------------- | ---------------- |
25+
| Rust list_directory | 308ms |
26+
| JSON serialize | 18ms |
27+
| IPC transfer (17.4 MB) | ~4,100ms |
28+
| JSON.parse | 67ms |
29+
| Svelte reactivity | ~9,500ms |
30+
| **Total** | **~14s** |
3031

31-
The user experiences a frozen UI during the Svelte reactivity step because assigning large arrays to reactive state triggers expensive internal tracking.
32+
The user experiences a frozen UI during the Svelte reactivity step because assigning large arrays to reactive state
33+
triggers expensive internal tracking.
3234

3335
---
3436

@@ -115,61 +117,62 @@ Time →
115117

116118
### Key design decisions
117119

118-
**FileDataStore per pane:**
119-
Each pane has its own store because:
120+
**FileDataStore per pane:** Each pane has its own store because:
121+
120122
- Panes may show the same folder but with different sorting/filtering
121123
- Simplifies state management
122124

123-
**Item ID = path (not filename):**
124-
Path is guaranteed unique and handles future cross-directory caching.
125+
**Item ID = path (not filename):** Path is guaranteed unique and handles future cross-directory caching.
125126

126-
**extendedMetadataLoaded flag:**
127-
Each FileEntry has a flag indicating whether extended metadata is loaded. UI can show placeholder or partial data until extended data arrives.
127+
**extendedMetadataLoaded flag:** Each FileEntry has a flag indicating whether extended metadata is loaded. UI can show
128+
placeholder or partial data until extended data arrives.
128129

129130
**Hidden files filtering:**
130-
Cannot know exact visible count until all files are loaded (some may be hidden). Handle gracefully—show what we have, update count as data arrives.
131+
Cannot know exact visible count until all files are loaded (some may be hidden). Handle gracefully—show what we have,
132+
update count as data arrives.
131133

132-
**Chunk size:**
133-
5000 files per chunk balances IPC overhead vs. responsiveness. May tune later.
134+
**Chunk size:** 5000 files per chunk balances IPC overhead vs. responsiveness. May tune later.
134135

135-
**Sorting placeholders:**
136-
Backend sorting (when implemented) happens before chunking so first chunk contains the "best" files for the current sort order.
136+
**Sorting placeholders:** Backend sorting (when implemented) happens before chunking so first chunk contains the "best"
137+
files for the current sort order.
137138

138139
---
139140

140141
## Cancellation
141142

142143
**Current state:**
144+
143145
- ✅ Frontend: `loadGeneration` counter discards stale results
144146
- ❌ Backend: Rust `list_directory()` runs to completion even if user navigates away
145147

146-
**Future optimization:** Add cancellation flag checked periodically in Rust loop. Not critical for UX since UI doesn't freeze.
148+
**Future optimization:** Add cancellation flag checked periodically in Rust loop. Not critical for UX since UI doesn't
149+
freeze.
147150

148151
---
149152

150153
## Key clues
151154

152155
Files and patterns to understand before implementing:
153156

154-
| File | Why it matters |
155-
|------|----------------|
156-
| `src/lib/file-explorer/FilePane.svelte` | Current loading logic lives here. Look for `loadDirectory()`, `allFilesRaw`, `filesVersion`. This is what you're replacing. |
157-
| `src/lib/file-explorer/BriefList.svelte` | Virtual scroll for Brief mode. Already calculates `startIndex`/`endIndex`. You'll modify it to request visible range from the store. |
158-
| `src/lib/file-explorer/FullList.svelte` | Virtual scroll for Full mode. Same pattern as BriefList. |
159-
| `src-tauri/src/file_system/operations.rs` | Rust session management (`list_directory_start`, `list_directory_next`, etc.). This is where streaming logic lives. |
160-
| `src-tauri/src/commands/file_system.rs` | Tauri command wrappers that call `operations.rs`. New commands must be added here. |
161-
| `src-tauri/src/lib.rs` | Command registration. New Rust commands must be registered in the `invoke_handler`. |
162-
| `src/lib/tauri-commands.ts` | Frontend TypeScript wrappers for Tauri commands. Add new command wrappers here. |
163-
| `src/lib/file-explorer/types.ts` | `FileEntry` type definition. Add `extendedMetadataLoaded` flag here. |
157+
| File | Why it matters |
158+
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
159+
| `src/lib/file-explorer/FilePane.svelte` | Current loading logic lives here. Look for `loadDirectory()`, `allFilesRaw`, `filesVersion`. This is what you're replacing. |
160+
| `src/lib/file-explorer/BriefList.svelte` | Virtual scroll for Brief mode. Already calculates `startIndex`/`endIndex`. You'll modify it to request visible range from the store. |
161+
| `src/lib/file-explorer/FullList.svelte` | Virtual scroll for Full mode. Same pattern as BriefList. |
162+
| `src-tauri/src/file_system/operations.rs` | Rust session management (`list_directory_start`, `list_directory_next`, etc.). This is where streaming logic lives. |
163+
| `src-tauri/src/commands/file_system.rs` | Tauri command wrappers that call `operations.rs`. New commands must be added here. |
164+
| `src-tauri/src/lib.rs` | Command registration. New Rust commands must be registered in the `invoke_handler`. |
165+
| `src/lib/tauri-commands.ts` | Frontend TypeScript wrappers for Tauri commands. Add new command wrappers here. |
166+
| `src/lib/file-explorer/types.ts` | `FileEntry` type definition. Add `extendedMetadataLoaded` flag here. |
164167

165168
**Where to put `FileDataStore`:** Create it at `src/lib/file-explorer/FileDataStore.ts`.
166169

167-
**Verification:** Run `./scripts/check.sh` after each phase to ensure nothing is broken (runs rustfmt, clippy, tests, eslint, svelte-check, etc.).
170+
**Verification:** Run `./scripts/check.sh` after each phase to ensure nothing is broken (runs rustfmt, clippy, tests,
171+
eslint, svelte-check, etc.).
168172

169173
---
170174

171175
## Related documents
172176

173177
- [ADR-009: Non-reactive file data store](../adr/009-non-reactive-file-store.md) — why we need FileDataStore
174178
- [2025-12-28-dir-load-bench-findings.md](./2025-12-28-dir-load-bench-findings.md) — benchmark data
175-

docs/workflows/documenting-key-tech-decisions.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,8 @@ A short summary of the context, problem, solution, and consequences. In clear la
2222

2323
### Context
2424

25-
Give all information that helps understand the background and reasoning behind the decision,
26-
everything that leads up to the problem statement, but not the problem itself.
25+
Give all information that helps understand the background and reasoning behind the decision, everything that leads up to
26+
the problem statement, but not the problem itself.
2727

2828
### Problem
2929

0 commit comments

Comments
 (0)