Skip to content

Commit b575cda

Browse files
Add module→path translator + iso-trace JSON ingestion adapter
Two independent improvements that compose: 1. Module→path translator in retrieve_brief's lexical fallback. Issue text in SWE-bench-Verified often references Python modules dotted-style (`astropy.modeling.separable_matrix`) rather than as file paths. Pre-fix, the fallback only mined bare paths + URLs and missed these. Now `_module_to_path_candidates` regex-matches 3+-segment all-lowercase dotted names and emits both `astropy/modeling/separable.py` and (when 4+ segments) the parent path. Conservative on capitalized segments to avoid pulling class references. On n=50 smoke: state_trace A@1 jumped 0.160 → 0.200, A@5 0.240 → 0.320 vs. bm25 unchanged at 0.120/0.260. Full n=500 re-run in progress; will update README headline when it lands. 2. state_trace/iso_trace_adapter.py — new ingestion adapter for @razroo/iso-trace's normalized Session → Turn → Event[] JSON shape. Reads the documented schema (no iso-trace dep), converts to state-trace's agent_log format, and ingests through the existing pipeline. Lets users with months of accumulated Claude Code / Cursor / Codex / OpenCode sessions seed state-trace working memory without re-running the agent. Tool-call → action mapping handles Edit/Read/Bash/Glob/Grep with action_kind inference; tool_result errors flag status=error and capture error_signatures; file_op events flow through to step.files. Two tests cover the pure transform and end-to-end engine ingestion. 52 tests passing (was 50). README documents the iso-trace ingestion path with the npx export-fixture command. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 0ff2240 commit b575cda

5 files changed

Lines changed: 517 additions & 2 deletions

File tree

README.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,25 @@ engine.store_agent_log_file("examples/data/agent_logs/marshmallow__marshmallow-1
257257

258258
Supported inputs: normalized `agent_log` JSON, raw SWE-agent `.traj` files, raw OpenHands event JSON logs.
259259

260+
### From iso-trace (Claude Code / Cursor / Codex / opencode sessions)
261+
262+
If you've accumulated session history with [`@razroo/iso-trace`](https://www.npmjs.com/package/@razroo/iso-trace), feed it directly:
263+
264+
```bash
265+
# Export a session via iso-trace's CLI
266+
npx @razroo/iso-trace export <session-id> --json --out session.json
267+
```
268+
269+
```python
270+
from state_trace import MemoryEngine
271+
from state_trace.iso_trace_adapter import ingest_iso_trace_session
272+
273+
engine = MemoryEngine(capacity_limit=256.0, namespace="my-repo")
274+
ingest_iso_trace_session(engine, "session.json")
275+
```
276+
277+
The adapter reads iso-trace's documented Session → Turn → Event[] JSON and converts it to state-trace's `agent_log` format — typed nodes for files, edits, tests, errors. Months of accumulated harness history become queryable working memory without re-running the agent.
278+
260279
## Live solve-rate (next credibility step)
261280

262281
`examples/swebench_verified_solve_rate.py` scaffolds end-to-end solve-rate measurement: state-trace brief → LLM patch proposal → SWE-bench-Verified prediction JSONL. It does not run the swebench docker harness; that step is documented in the script's header.

state_trace/iso_trace_adapter.py

Lines changed: 242 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,242 @@
1+
"""Adapter for ingesting iso-trace session JSON into state-trace.
2+
3+
`iso-trace` (https://www.npmjs.com/package/@razroo/iso-trace) parses Claude
4+
Code, Cursor, Codex, and OpenCode transcripts into a normalized
5+
Session → Turn → Event[] JSON shape. This adapter converts that JSON into
6+
state-trace's `agent_log` ingestion format and feeds it through the
7+
existing typed-node + edge pipeline, so users with months of accumulated
8+
session history can reuse it as state-trace working memory without
9+
re-running the agent.
10+
11+
Schema reference (excerpt from @razroo/iso-trace's types.d.ts):
12+
13+
Session: { id, source, cwd, model, durationMs, turns, tokenUsage }
14+
Turn: { index, role, at, events }
15+
Event: message | tool_call | tool_result | file_op | token_usage
16+
17+
Usage:
18+
19+
from state_trace import MemoryEngine
20+
from state_trace.iso_trace_adapter import ingest_iso_trace_session
21+
22+
engine = MemoryEngine(capacity_limit=256.0, namespace="my-repo")
23+
ingest_iso_trace_session(engine, "/path/to/exported-session.json")
24+
25+
The adapter does not import anything from iso-trace itself — it operates
26+
on the documented JSON shape, so it works against any export iso-trace
27+
produces (and any compatible third-party emitter).
28+
"""
29+
30+
from __future__ import annotations
31+
32+
import json
33+
from pathlib import Path
34+
from typing import Any
35+
36+
from state_trace.utils import classify_observation_status, infer_action_kind
37+
38+
39+
def ingest_iso_trace_session(
40+
engine: Any,
41+
path: str | Path,
42+
context: dict[str, Any] | None = None,
43+
) -> dict[str, Any]:
44+
"""Read an iso-trace session JSON file and ingest it into the engine.
45+
46+
Returns the normalized agent_log dict that was ingested (useful for
47+
inspection / debugging). The actual graph mutation happens via
48+
`engine.store_agent_log(...)`.
49+
"""
50+
session = json.loads(Path(path).read_text(encoding="utf-8"))
51+
normalized = iso_trace_session_to_agent_log(session, default_context=context)
52+
engine.store_agent_log(normalized, context=context)
53+
return normalized
54+
55+
56+
def iso_trace_session_to_agent_log(
57+
session: dict[str, Any],
58+
default_context: dict[str, Any] | None = None,
59+
) -> dict[str, Any]:
60+
"""Convert an iso-trace Session JSON dict to state-trace's agent_log format.
61+
62+
Pure transform — no graph mutation, no I/O. Useful for tests and for
63+
callers that want to inspect or post-process the normalized form before
64+
ingestion.
65+
"""
66+
default_context = default_context or {}
67+
turns = session.get("turns", []) or []
68+
69+
issue_title, issue_text = _extract_issue(turns)
70+
repo = _extract_repo(session)
71+
session_id = str(session.get("id") or default_context.get("session") or "iso-trace-session")
72+
goal = str(default_context.get("goal") or f"resume from {session_id}")
73+
74+
steps: list[dict[str, Any]] = []
75+
for index, turn in enumerate(turns):
76+
step = _turn_to_step(turn, index)
77+
if step is not None:
78+
steps.append(step)
79+
80+
return {
81+
"format": "agent_log",
82+
"source": "iso_trace",
83+
"source_url": _extract_source_url(session),
84+
"session_id": session_id,
85+
"issue_title": issue_title,
86+
"issue_text": issue_text,
87+
"goal": goal,
88+
"repo": repo,
89+
"steps": steps,
90+
"submission_diff": "",
91+
"submission_files": [],
92+
}
93+
94+
95+
def _extract_issue(turns: list[dict[str, Any]]) -> tuple[str, str]:
96+
"""First user message becomes the issue title + text."""
97+
for turn in turns:
98+
for event in turn.get("events", []) or []:
99+
if event.get("kind") == "message" and event.get("role") == "user":
100+
text = str(event.get("text", "")).strip()
101+
if text:
102+
title = text.split("\n", 1)[0][:160]
103+
return title, text
104+
return "(no user prompt)", ""
105+
106+
107+
def _extract_repo(session: dict[str, Any]) -> str | None:
108+
"""Best-effort: derive a repo identifier from cwd basename."""
109+
cwd = session.get("cwd")
110+
if not cwd:
111+
return None
112+
return Path(str(cwd)).name or None
113+
114+
115+
def _extract_source_url(session: dict[str, Any]) -> str | None:
116+
src = session.get("source") or {}
117+
if not isinstance(src, dict):
118+
return None
119+
return src.get("path")
120+
121+
122+
def _turn_to_step(turn: dict[str, Any], index: int) -> dict[str, Any] | None:
123+
"""Squash one Turn's events into one agent_log step.
124+
125+
Mapping (lossy but functional):
126+
- assistant `message` -> step.thought
127+
- first `tool_call` -> step.action + action_kind
128+
- first `tool_result` -> step.observation + status
129+
- all `file_op` -> step.files
130+
- tool_result.error -> error signature
131+
132+
Turns with only message events still produce a step (thought-only).
133+
Turns with no actionable content (e.g. system-only) are skipped.
134+
"""
135+
events = turn.get("events", []) or []
136+
if not events:
137+
return None
138+
139+
thought_parts: list[str] = []
140+
action: str = ""
141+
action_kind: str | None = None
142+
observation: str = ""
143+
status: str | None = None
144+
files: list[str] = []
145+
error_signatures: list[str] = []
146+
seen_tool_call = False
147+
148+
for event in events:
149+
kind = event.get("kind")
150+
if kind == "message":
151+
role = event.get("role")
152+
text = str(event.get("text", "")).strip()
153+
if not text:
154+
continue
155+
if role == "assistant":
156+
thought_parts.append(text)
157+
elif kind == "tool_call":
158+
if seen_tool_call:
159+
continue
160+
seen_tool_call = True
161+
tool_name = str(event.get("name", "")).strip()
162+
tool_input = event.get("input")
163+
action = _format_tool_call(tool_name, tool_input)
164+
action_kind = _infer_action_kind(tool_name, tool_input, action)
165+
elif kind == "tool_result":
166+
output = str(event.get("output", "")).strip()
167+
err = event.get("error")
168+
if err:
169+
observation = output or str(err)
170+
status = "error"
171+
error_signatures.append(str(err)[:240])
172+
else:
173+
observation = output
174+
if not status:
175+
status = classify_observation_status(observation)
176+
elif kind == "file_op":
177+
path = event.get("path")
178+
if path and path not in files:
179+
files.append(str(path))
180+
181+
# Skip turns with no actionable content (e.g. pure system messages).
182+
if not (action or observation or thought_parts or files):
183+
return None
184+
185+
step: dict[str, Any] = {
186+
"index": index,
187+
"action": action,
188+
"thought": "\n".join(thought_parts).strip(),
189+
"observation": observation,
190+
"files": files,
191+
"action_kind": action_kind or "message",
192+
"status": status or "info",
193+
"error_signatures": error_signatures,
194+
}
195+
return step
196+
197+
198+
def _format_tool_call(name: str, input_value: Any) -> str:
199+
"""Render a tool_call into the action-string shape state-trace expects.
200+
201+
Examples:
202+
("Edit", {"file_path": "src/auth.ts"}) -> 'edit "src/auth.ts"'
203+
("Bash", {"command": "pytest tests/"}) -> 'pytest tests/'
204+
("Read", {"file_path": "src/x.py"}) -> 'open "src/x.py"'
205+
"""
206+
if not name:
207+
return ""
208+
lowered = name.lower()
209+
if isinstance(input_value, dict):
210+
# Common file-touching tool fields
211+
for key in ("file_path", "path", "filename"):
212+
if key in input_value:
213+
file_arg = str(input_value[key])
214+
if lowered in {"edit", "write", "multiedit"}:
215+
return f'edit "{file_arg}"'
216+
if lowered in {"read", "view"}:
217+
return f'open "{file_arg}"'
218+
if lowered == "glob":
219+
return f"find_file {file_arg}"
220+
return f'{lowered} "{file_arg}"'
221+
if "command" in input_value:
222+
return str(input_value["command"])
223+
if "query" in input_value:
224+
return f'{lowered}: {input_value["query"]}'
225+
return f"{lowered}: {json.dumps(input_value)[:200] if input_value else ''}".strip(": ")
226+
227+
228+
def _infer_action_kind(tool_name: str, input_value: Any, rendered_action: str) -> str:
229+
"""Map iso-trace tool names + rendered actions to state-trace's action_kind."""
230+
lowered = tool_name.lower()
231+
if lowered in {"edit", "write", "multiedit"}:
232+
return "edit"
233+
if lowered in {"read", "view"}:
234+
return "open"
235+
if lowered == "glob":
236+
return "find_file"
237+
if lowered == "grep":
238+
return "search"
239+
if lowered == "bash":
240+
# Use shared heuristic to infer pytest/python/etc. from the command.
241+
return infer_action_kind(rendered_action)
242+
return lowered or "message"

state_trace/retrieval.py

Lines changed: 60 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1314,16 +1314,69 @@ def _brief_file_priority(node: dict[str, Any], profile: dict[str, Any]) -> float
13141314
r"https?://(?:www\.)?gitlab\.com/[^/\s]+/[^/\s]+/-/blob/[^/\s]+/([^\s)#?]+)",
13151315
re.IGNORECASE,
13161316
)
1317+
# Python module references like `astropy.modeling.separable` or
1318+
# `django.utils.html.escape` mentioned in issue text. The trailing segment
1319+
# could be a function, class, or submodule — we generate path candidates
1320+
# for both interpretations.
1321+
_PYTHON_MODULE_RE = re.compile(
1322+
r"\b([a-z][a-z0-9_]*(?:\.[a-z_][a-z0-9_]*){2,})\b"
1323+
)
1324+
1325+
1326+
def _module_to_path_candidates(text: str) -> list[str]:
1327+
"""Translate Python module references in `text` into file path candidates.
1328+
1329+
`astropy.modeling.separable` produces both:
1330+
- `astropy/modeling/separable.py` (the dotted name is a module)
1331+
- `astropy/modeling.py` + `astropy/modeling/separable.py` is also
1332+
plausible if the LAST segment is a function or class — we hedge by
1333+
also emitting the parent path when the last segment looks lowercase
1334+
(likely a function or submodule, not a class).
1335+
1336+
Conservative: only matches dotted names with at least 3 segments, all
1337+
lowercase (Python class names start uppercase, which we skip). Common
1338+
stdlib namespaces like `os.path` are short enough not to match.
1339+
"""
1340+
seen: set[str] = set()
1341+
out: list[str] = []
1342+
for match in _PYTHON_MODULE_RE.findall(text):
1343+
segments = match.split(".")
1344+
# Skip if any segment looks like a class (capitalized) — those are
1345+
# type references, not module paths. PYTHON_MODULE_RE is already
1346+
# lowercase-only via the character class, but be defensive.
1347+
if any(seg and seg[0].isupper() for seg in segments):
1348+
continue
1349+
# Skip well-known non-module dotted names that are technical jargon.
1350+
joined = match.lower()
1351+
if joined.startswith(("e.g", "i.e", "vs.")) or joined in {"a.b.c"}:
1352+
continue
1353+
# Module-as-file: astropy.modeling.separable -> astropy/modeling/separable.py
1354+
candidate = "/".join(segments) + ".py"
1355+
if candidate not in seen:
1356+
seen.add(candidate)
1357+
out.append(candidate)
1358+
# Submodule-as-file (drop the final segment if it's plausibly a
1359+
# function or class name on the parent module):
1360+
# astropy.modeling.separable.separability_matrix
1361+
# -> astropy/modeling/separable.py
1362+
if len(segments) >= 4:
1363+
parent_candidate = "/".join(segments[:-1]) + ".py"
1364+
if parent_candidate not in seen:
1365+
seen.add(parent_candidate)
1366+
out.append(parent_candidate)
1367+
return out
13171368

13181369

13191370
def _extract_lexical_file_candidates(text: str) -> list[str]:
13201371
"""Mine repo-relative file paths from text, including ones embedded in
1321-
code-host URLs.
1372+
code-host URLs and inferred from Python module references.
13221373
13231374
Normal trajectory ingestion uses `extract_file_paths`, which rejects URLs
13241375
(they're rarely meaningful during edit/test loops). At cold start from an
13251376
issue text, the golden file is far more often embedded in a github.com
1326-
blob URL than bare, so the fallback path mines both.
1377+
blob URL than bare, AND issue authors describe the bug location using
1378+
dotted Python module names (`astropy.modeling.separable_matrix`) instead
1379+
of repo-relative paths. The fallback mines all three forms.
13271380
"""
13281381
candidates = list(extract_file_paths(text))
13291382
seen = {candidate for candidate in candidates}
@@ -1334,6 +1387,11 @@ def _extract_lexical_file_candidates(text: str) -> list[str]:
13341387
continue
13351388
candidates.append(path)
13361389
seen.add(path)
1390+
for module_path in _module_to_path_candidates(text):
1391+
if module_path in seen:
1392+
continue
1393+
candidates.append(module_path)
1394+
seen.add(module_path)
13371395
return candidates
13381396

13391397

0 commit comments

Comments
 (0)