@@ -25,7 +25,7 @@ Recursive workflows enable complex, multi-level task execution with built-in sta
2525
2626** Hierarchical State Tracking** : Build multi-level task trees with parent-child relationships, enabling complex workflow decomposition and progress tracking.
2727
28- ** Persistent State** : Task state persists across workflow calls in JSON files , enabling pause/resume, progress monitoring, and audit trails.
28+ ** Persistent State** : Task state persists across workflow calls in SQLite databases , enabling pause/resume, progress monitoring, and audit trails with concurrent access support .
2929
3030** Progress Aggregation** : Automatic rollup of child task status, including completion percentages, pending tasks, and failure detection.
3131
@@ -153,7 +153,7 @@ result = await execute_workflow(
153153 }
154154)
155155
156- # State file created at: ~/.workflows/tasks/<trace_id>.json
156+ # State database created at: ~/.workflows/tasks/<trace_id>.db
157157print(f"State : {result['state']}")
158158print(f"Processed : {result['processed_count']} items")
159159` ` `
@@ -180,42 +180,71 @@ await execute_workflow(
180180
181181## State Management Basics
182182
183- ### State File Structure
183+ ### State Database Structure
184+
185+ The state management system persists task hierarchies in SQLite databases at ` ~/.workflows/tasks/<trace_id>.db ` .
186+
187+ ** Database Schema:**
188+
189+ ``` sql
190+ -- Task hierarchy with parent-child relationships
191+ CREATE TABLE tasks (
192+ task_id TEXT PRIMARY KEY ,
193+ parent_id TEXT REFERENCES tasks(task_id),
194+ task TEXT NOT NULL ,
195+ task_type TEXT DEFAULT ' ' ,
196+ status TEXT DEFAULT ' pending' ,
197+ data JSON DEFAULT ' {}' ,
198+ created_at TEXT NOT NULL ,
199+ updated_at TEXT NOT NULL
200+ );
201+
202+ -- Key-value memory storage for workflow state
203+ CREATE TABLE memory (
204+ key TEXT PRIMARY KEY ,
205+ value JSON,
206+ updated_at TEXT NOT NULL
207+ );
208+
209+ -- Audit trail for all state operations
210+ CREATE TABLE audit (
211+ id INTEGER PRIMARY KEY AUTOINCREMENT,
212+ timestamp TEXT NOT NULL ,
213+ task_id TEXT ,
214+ caller TEXT NOT NULL ,
215+ action TEXT NOT NULL ,
216+ description TEXT ,
217+ changes JSON,
218+ parent_id TEXT
219+ );
220+
221+ -- Iteration tracking for loops
222+ CREATE TABLE iterations (
223+ task_id TEXT PRIMARY KEY ,
224+ current INTEGER DEFAULT 0 ,
225+ total INTEGER DEFAULT 0 ,
226+ cap INTEGER DEFAULT 100 ,
227+ started_at TEXT ,
228+ completed_at TEXT ,
229+ checkpoints JSON DEFAULT ' []'
230+ );
231+ ```
184232
185- The state management system persists task hierarchies in JSON files at ` ~/.workflows/tasks/<trace_id>.json ` :
233+ ** Example task data (logical structure): **
186234
187235``` json
188236{
189- "root_task_id" : " task-abc123" ,
237+ "task_id" : " task-abc123" ,
238+ "task" : " PR Review" ,
239+ "task_type" : " pr-review" ,
240+ "status" : " in-progress" ,
241+ "parent_id" : null ,
242+ "data" : {
243+ "platform" : " github" ,
244+ "files_changed" : 19
245+ },
190246 "created_at" : " 2025-12-12T10:30:00.123Z" ,
191- "updated_at" : " 2025-12-12T10:35:45.456Z" ,
192- "tasks" : {
193- "task-abc123" : {
194- "task" : " PR Review" ,
195- "task_type" : " pr-review" ,
196- "status" : " in-progress" ,
197- "parent_id" : null ,
198- "children" : [" task-def456" , " task-ghi789" ],
199- "data" : {
200- "platform" : " github" ,
201- "files_changed" : 19
202- },
203- "created_at" : " 2025-12-12T10:30:00.123Z" ,
204- "updated_at" : " 2025-12-12T10:35:45.456Z"
205- },
206- "task-def456" : {
207- "task" : " Phase: Context Gathering" ,
208- "task_type" : " phase" ,
209- "status" : " done" ,
210- "parent_id" : " task-abc123" ,
211- "children" : [],
212- "data" : {
213- "files_found" : 19
214- },
215- "created_at" : " 2025-12-12T10:30:01.234Z" ,
216- "updated_at" : " 2025-12-12T10:30:11.234Z"
217- }
218- }
247+ "updated_at" : " 2025-12-12T10:35:45.456Z"
219248}
220249```
221250
@@ -234,7 +263,7 @@ The state management system persists task hierarchies in JSON files at `~/.workf
234263The ` agent-state-management ` workflow provides comprehensive outputs for building recursive patterns:
235264
236265#### Core Identifiers
237- - ** ` state ` ** : Path to state file (` ~/.workflows/tasks/<trace_id>.json ` )
266+ - ** ` state ` ** : Path to SQLite database (` ~/.workflows/tasks/<trace_id>.db ` )
238267- ** ` task_id ` ** : ID of created/updated task
239268- ** ` root_task_id ` ** : ID of the root task in the tree
240269- ** ` trace_id ` ** : Execution trace ID (filename without extension)
@@ -265,38 +294,42 @@ The `agent-state-management` workflow provides comprehensive outputs for buildin
265294
266295### Audit Trail
267296
268- Each state operation is logged to a separate audit file (` <trace_id>.audit.json ` ):
297+ Each state operation is logged to the ` audit ` table in the same SQLite database. Query with SQL:
298+
299+ ``` sql
300+ SELECT timestamp , task_id, caller, action, description, changes
301+ FROM audit
302+ ORDER BY timestamp ;
303+ ```
304+
305+ ** Example audit entries (logical structure):**
269306
270307``` json
271- {
272- "entries" : [
273- {
274- "timestamp" : " 2025-12-12T10:30:00.123Z" ,
275- "task_id" : " task-abc123" ,
276- "caller" : " pr-review" ,
277- "action" : " create_root" ,
278- "description" : " Created root task: PR Review"
279- },
280- {
281- "timestamp" : " 2025-12-12T10:30:01.234Z" ,
282- "task_id" : " task-def456" ,
283- "parent_id" : " task-abc123" ,
284- "caller" : " pr-review:context-gathering" ,
285- "action" : " create_subtask" ,
286- "description" : " Created sub-task: Phase: Context Gathering"
287- },
288- {
289- "timestamp" : " 2025-12-12T10:30:11.234Z" ,
290- "task_id" : " task-def456" ,
291- "caller" : " pr-review:context-gathering" ,
292- "action" : " task_completed" ,
293- "description" : " Updated: ['status']" ,
294- "changes" : {
295- "status" : [" in-progress" , " done" ]
296- }
297- }
298- ]
299- }
308+ [
309+ {
310+ "timestamp" : " 2025-12-12T10:30:00.123Z" ,
311+ "task_id" : " task-abc123" ,
312+ "caller" : " pr-review" ,
313+ "action" : " create_root" ,
314+ "description" : " Created root task: PR Review"
315+ },
316+ {
317+ "timestamp" : " 2025-12-12T10:30:01.234Z" ,
318+ "task_id" : " task-def456" ,
319+ "parent_id" : " task-abc123" ,
320+ "caller" : " pr-review:context-gathering" ,
321+ "action" : " create_subtask" ,
322+ "description" : " Created sub-task: Phase: Context Gathering"
323+ },
324+ {
325+ "timestamp" : " 2025-12-12T10:30:11.234Z" ,
326+ "task_id" : " task-def456" ,
327+ "caller" : " pr-review:context-gathering" ,
328+ "action" : " task_completed" ,
329+ "description" : " Updated: ['status']" ,
330+ "changes" : {"status" : [" in-progress" , " done" ]}
331+ }
332+ ]
300333```
301334
302335---
@@ -1039,7 +1072,7 @@ Use the `agent-state-visualize` workflow to inspect task hierarchies:
10391072result = await execute_workflow(
10401073 workflow="agent-state-visualize",
10411074 inputs={
1042- "state": "/Users/user/.workflows/tasks/exec-abc123.json ",
1075+ "state": "/Users/user/.workflows/tasks/exec-abc123.db ",
10431076 "show_data": True,
10441077 "max_depth": 0 # 0 = unlimited
10451078 }
@@ -1092,40 +1125,45 @@ inputs:
10921125 description: Visualize subtree starting from this task
10931126` ` `
10941127
1095- # ## Reading State Files Directly
1128+ # ## Querying State Directly
10961129
1097- State files are human-readable JSON :
1130+ State is stored in SQLite with queryable tables :
10981131
10991132` ` ` bash
1100- # View state file
1101- cat ~/.workflows/tasks/exec-abc123.json | jq .
1133+ # Open database
1134+ sqlite3 ~/.workflows/tasks/exec-abc123.db
11021135
11031136# List all tasks
1104- cat ~/.workflows/tasks/exec-abc123.json | jq '.tasks | keys'
1137+ SELECT task_id, status, task FROM tasks;
11051138
11061139# Get task details
1107- cat ~/.workflows/ tasks/exec-abc123.json | jq '.tasks[" task-abc123"]'
1140+ SELECT * FROM tasks WHERE task_id = ' task-abc123';
11081141
11091142# Check children status
1110- cat ~/.workflows/tasks/exec-abc123.json | \
1111- jq '.tasks["task-abc123"].children | map(.status)'
1143+ SELECT task_id, status FROM tasks WHERE parent_id = 'task-abc123';
1144+
1145+ # Task tree with hierarchy
1146+ SELECT t.task_id, t.status, t.task, p.task_id as parent
1147+ FROM tasks t
1148+ LEFT JOIN tasks p ON t.parent_id = p.task_id;
11121149` ` `
11131150
11141151# ## Audit Trail Analysis
11151152
11161153Review audit logs to understand workflow execution :
11171154
11181155` ` ` bash
1156+ # Open database
1157+ sqlite3 ~/.workflows/tasks/exec-abc123.db
1158+
11191159# View audit log
1120- cat ~/.workflows/tasks/exec-abc123. audit.json | jq .
1160+ SELECT * FROM audit ORDER BY timestamp;
11211161
11221162# Filter by caller
1123- cat ~/.workflows/tasks/exec-abc123.audit.json | \
1124- jq '.entries[] | select(.caller == "pr-review:investigation-loop")'
1163+ SELECT * FROM audit WHERE caller LIKE 'pr-review:investigation-loop%';
11251164
11261165# Timeline of events
1127- cat ~/.workflows/tasks/exec-abc123.audit.json | \
1128- jq '.entries[] | {timestamp, task_id, action, description}'
1166+ SELECT timestamp, task_id, action, description FROM audit ORDER BY timestamp;
11291167` ` `
11301168
11311169---
@@ -1843,7 +1881,8 @@ execute_workflow(
18431881
18441882` ` ` bash
18451883# Check parent-child relationships
1846- cat ~/.workflows/tasks/exec-abc123.json | jq '.tasks | to_entries | .[] | {id: .key, parent: .value.parent_id, children: .value.children}'
1884+ sqlite3 ~/.workflows/tasks/exec-abc123.db \
1885+ "SELECT task_id, parent_id FROM tasks"
18471886` ` `
18481887
18491888**Solution**: Ensure `parent_id` matches the actual parent's `task_id`.
@@ -1854,7 +1893,7 @@ cat ~/.workflows/tasks/exec-abc123.json | jq '.tasks | to_entries | .[] | {id: .
18541893
18551894# ## Q: Can I pause and resume recursive workflows?
18561895
1857- **A**: Yes, the state file persists between calls. You can stop execution and resume later by passing the same `state` path:
1896+ **A**: Yes, the SQLite database persists between calls. You can stop execution and resume later by passing the same `state` path:
18581897
18591898` ` ` yaml
18601899# First execution
@@ -1879,13 +1918,13 @@ result2 = execute_workflow(workflow="my-workflow", inputs={"items": [...], "stat
18791918 for_each_mode: parallel # Process all files concurrently
18801919` ` `
18811920
1882- # ## Q: How do I delete old state files ?
1921+ # ## Q: How do I delete old state databases ?
18831922
1884- **A**: State files accumulate in `~/.workflows/tasks/`. Clean up manually:
1923+ **A**: State databases accumulate in `~/.workflows/tasks/`. Clean up manually:
18851924
18861925` ` ` bash
1887- # Delete state files older than 7 days
1888- find ~/.workflows/tasks -name "*.json " -mtime +7 -delete
1926+ # Delete state databases older than 7 days
1927+ find ~/.workflows/tasks -name "*.db " -mtime +7 -delete
18891928` ` `
18901929
18911930# ## Q: Can I use state management without recursion?
@@ -1894,7 +1933,7 @@ find ~/.workflows/tasks -name "*.json" -mtime +7 -delete
18941933
18951934# ## Q: What's the performance impact of state tracking?
18961935
1897- **A**: Minimal - state operations are fast file writes . Typical overhead: < 100ms per task.
1936+ **A**: Minimal - SQLite operations are fast with WAL mode . Typical overhead: < 100ms per task. Concurrent writes are handled automatically .
18981937
18991938# ## Q: Can I share state between different workflows?
19001939
0 commit comments