Python: Pipfile support and dependency recipe rework (#7521)

* Added pipenv lockfiles

* Added pip parsing

* Added pip parsing

* Added pip parsing

* Plan: Python dependency recipe Accumulator rework

* Add PyProjectHelper.regenerateLockContent dispatcher

* AddDependency: move per-project state to Accumulator and run regen in generate()

Refactors AddDependency to use a richer Accumulator with per-project
ProjectState objects that capture lock file paths, lock content, and
dependency-match flags during scanning. Dependency modification and
lock regeneration now run in the visitor on the live (chain-modified)
tree so that composite recipes correctly chain their modifications.

Replaces ExecutionContext-based lock state with accumulator-based state,
which keeps per-recipe state isolated and makes the approach composable.

* AddDependency: move per-project state to Accumulator and run regen in visitor

Replaces the PythonDependencyExecutionContextView-based state with per-project
ProjectState entries on the Accumulator. The scanner captures lock-file content
and the recipe-specific match predicate; the visitor applies the trait edit on
the live tree, refreshes the marker, runs lock regeneration, and caches the
result. The lock-file visit reads the cache and re-emits the regenerated
content. This composes correctly across CompositeRecipe chains because each
recipe's edit lands before the next recipe's edit phase begins.

* AddDependency: ctx side-channel for cross-recipe deps tree sync

Composite chains of dependency recipes need the lock-file branch to see
the latest deps tree (with prior recipe edits applied), not the original
captured by the scanner. The scanner runs on originals before any edit
phase, so a B that follows A would otherwise regenerate from A's pre-edit
input.

Move per-source compute into the visitor's pre-visit and use an
ExecutionContext side channel keyed by deps-file path:
- deps-file branch: trait edit on the live cursor, write the modified
  tree to the ctx side channel, regenerate lock content if scanner
  captured one for this project.
- lock-file branch: lazily compute on first visit by reading the live
  deps tree from ctx (or the scanner-captured fallback when no prior
  recipe touched this path), reapplying the trait edit through a
  synthetic Cursor rooted at the visitor's root cursor.

This works across composite recipe chains because the side channel is
ExecutionContext-scoped, and tolerates within-cycle visit-order
variations because the lock branch lazy-computes whenever it visits
first.

* AddDependency / PyProjectHelper: post-prototype simplifications

Cross-confirmed findings from /simplify review:

- Drop unused ProjectState.lockFilePath (set but never read).
- Drop overridden generate() that returned an empty list — it's the
  ScanningRecipe default.
- Drop the parallel RegenerationResult value type — collapse onto
  LockFileRegeneration.Result, which has the same shape (success flag,
  lock content, error message). PyProjectHelper.regenerateLockContent
  now returns LockFileRegeneration.Result directly.
- Move the PackageManager → adapter switch onto LockFileRegeneration
  itself (LockFileRegeneration.forPackageManager(pm)). PyProjectHelper
  no longer encodes the UV/PIPENV mapping.
- Replace synthetic-cursor workaround new Cursor(getCursor().getRoot(),
  depsTree) with the idiomatic two-level form used by
  TraitMatcher.lower(SourceFile): new Cursor(new Cursor(null,
  Cursor.ROOT_VALUE), depsTree). Decouples the lock-branch trait match
  from the visitor's own cursor state.
- Reuse the visitor's matcher field on the lock branch instead of
  allocating a fresh PythonDependencyFile.Matcher per visit.
- Hoist the depsPath null-check and lockPs lookup with early returns,
  reducing nesting in the lock branch.
- Drop narrative comments that restate obvious code.
- Tighten imports in PyProjectHelper (use HashMap import, drop
  java.util.function.Function FQN where the import already exists).

* AddDependency / PyProjectHelper: collapse regen state, mark old API deprecated

Post-review cleanup before mirroring to RemoveDependency / ChangeDependency
/ UpgradeDependencyVersion / UpgradeTransitiveDependencyVersion:

- Collapse ProjectState.regeneratedLockContent + regenerationError into a
  single nullable LockFileRegeneration.Result. The two fields were
  mutually exclusive by construction (editAndRegenerate populates one or
  the other on a regen attempt) — keeping them parallel rebuilt the same
  redundancy that the previous /simplify pass already removed from
  EditAndRegenerateResult. Same change applied to EditAndRegenerateResult
  itself: the changed(modified, regen, error) factory is now changed(
  modified, @nullable Result). The visitor reads
  ps.regenResult.isSuccess() / getLockFileContent() / getErrorMessage()
  directly. The ensureComputed idempotence guard simplifies to a single
  modifiedDepsFile null-check (the regenerationError half was dead since
  modifiedDepsFile is also set in the error branch).

- Mark the legacy ExecutionContext-based PyProjectHelper helpers
  @deprecated: captureExistingLockContent, maybeReplayLockContent,
  maybeUpdateUvLock, maybeUpdatePipfileLock, regenerateLockAndRefreshMarker,
  regeneratePipfileLockAndRefreshMarker. They will be deleted in Task 8 of
  the Accumulator rework, but until then the @deprecated marker prevents
  the four sibling-recipe rewrites from accidentally reaching for them.

- Sync the plan's ProjectState shape with the implementation.

* Plan: sync Phase responsibilities with simplified ProjectState shape

* RemoveDependency: ctx side-channel for cross-recipe deps tree sync

* ChangeDependency: ctx side-channel for cross-recipe deps tree sync

* Plan: fix Task 4 template (ChangeDependency has no scope/groupName)

* UpgradeDependencyVersion: ctx side-channel for cross-recipe deps tree sync

* UpgradeTransitiveDependencyVersion: ctx side-channel for cross-recipe deps tree sync

* Remove afterModification from PythonDependencyFile trait

* Drop PythonDependencyExecutionContextView and unused PyProjectHelper helpers

* Remove internal planning doc

* Note in dependency-recipe descriptions that they aren't safe as preconditions

* Refresh PythonResolutionResult resolved deps after recipe edits

Recipes that edit a deps file and successfully regenerate its lock file
now overlay the resolved-dependency information from the regenerated
lock content onto the source file's PythonResolutionResult marker.
Previously editAndRegenerate only refreshed the declared-dep half of
the marker, so any subsequent recipe (or downstream consumer) reading
resolved dependencies still saw the pre-edit lock state.

Implementation:
- Extract the applyResolution + linkResolved + linkResolvedMap helpers
  shared by PyProjectTomlParser and PipfileParser into a new
  PythonResolutionLinker utility with two entry points
  (applyPyproject / applyPipfile) covering the differing sets of
  marker fields each format exposes.
- PyProjectHelper.applyResolvedDependencies(SourceFile, String)
  dispatches on the marker's PackageManager (Uv -> UvLockParser,
  Pipenv -> PipfileLockParser) to parse the lock content and run
  it through the linker.
- editAndRegenerate calls this overlay step after a successful regen,
  so the modifiedDepsFile returned to recipes carries a fully up-to-
  date marker.
- AddDependencyTest gains a marker-overlay assertion: after adding
  flask to a uv project, the post-recipe pyproject's marker contains
  flask in resolvedDependencies and the declared flask Dependency is
  linked to its ResolvedDependency entry.

No recipe-level changes; the helper boundary absorbs the new behavior.

* TomlParserVisitor: strip quotes from quoted-key identifier name

`Toml.Identifier.name` previously held the verbatim source for both bare
and quoted simple keys, so `"foo"` and `foo` produced different `name`
values even though the TOML spec treats them as the same key. Consumers
that matched on `getName()` had to strip quotes themselves at every site.

The class already has a separate `source` field for round-trip fidelity;
populate `name` with the unquoted form for simple keys while keeping
`source` as-is so the printer still emits the original.

Dotted keys are out of scope for this change. Without a dedicated
multi-segment AST type the parser cannot tell `site."google.com"` (two
segments, the second containing a literal dot) apart from `site.google.com`
(three segments) once both are flattened into a single `name` string, so
quoted segments inside a dotted key remain unstripped for now.

* Python deps recipes: handle quoted TOML keys and fix within-cycle chaining

Two bugs surfaced when running the dependency recipes via the Moderne
CLI against real Pipenv repositories.

**Quoted keys in `[packages]` / `[tool.pdm.overrides]`**

`Pipfile` and `pyproject.toml` allow quoted keys (e.g.
`"flake8" = "*"`). The recipes' match logic compared the package name
to `Toml.Identifier.getName()`, which now (with the rewrite-toml fix)
returns the unquoted form, so the comparisons just work. Removed the
ad-hoc instanceof+cast pattern from the call sites in favour of a
single null-safe `PyProjectHelper.extractKeyName(KeyValue)` helper used
by `PipfileFile`, `PipfileParser`, `PythonDependencyParser`, and
`PyProjectFile.upgradePdmOverride`. The PDM-overrides path also
silently ignored quoted keys via the same defect.

**Within-cycle chaining of recipes in a composite `recipeList`**

OpenRewrite's `RecipeRunCycle` runs every sub-recipe's scanner against
the original tree per cycle; only the edit phase chains within a
cycle. Each recipe was caching the scan-time `depsFileMatches` decision
and short-circuiting the visitor on it, so a downstream recipe in a
composite chain could not see additions/edits made by an earlier
recipe in the same cycle — convergence stretched across multiple
cycles, and lock-file regeneration ran on stale assumptions.

Drop the `depsFileMatches` field and the
`acc.projects.values().stream().noneMatch(...)` short-circuit in all 5
dependency recipes. Match decisions now happen at visit time against
the live trait obtained from the cursor (or via a synthetic cursor for
the lock-file lookahead path), so a chain like
`AddDependency → UpgradeDependencyVersion` for the same package
converges in a single cycle.

Tests:
- TOML-quoted-key coverage on all 5 recipe test classes (Pipfile and
  PDM-overrides cases).
- New `ChangeDependencyTest.chainAddThenUpgradeAcrossRecipes` exercises
  the within-cycle chain via a YAML composite recipe and asserts
  single-cycle convergence.

---------

Co-authored-by: Knut Wannheden <knut@moderne.io>

Author: Jenson3210

rewrite-json/src/main/java/org/openrewrite/json/JsonParser.java

@@ -78,7 +78,8 @@ public Stream<SourceFile> parse(@Language("json") String... sources) {
 
     @Override
     public boolean accept(Path path) {
-        return path.toString().endsWith(".json");
+        String fileName = path.getFileName().toString();
+        return fileName.endsWith(".json") || "Pipfile.lock".equals(fileName);
     }
 
     @Override

rewrite-python/build.gradle.kts

@@ -24,6 +24,7 @@ dependencies {
     api(project(":rewrite-core"))
     api(project(":rewrite-java"))
     api(project(":rewrite-toml"))
+    implementation(project(":rewrite-json"))
 
     api("org.jetbrains:annotations:latest.release")
     api("com.fasterxml.jackson.core:jackson-annotations")
@@ -126,6 +127,7 @@ testing {
             dependencies {
                 implementation(project())
                 implementation(project(":rewrite-java-21"))
+                implementation(project(":rewrite-json"))
                 implementation(project(":rewrite-test"))
                 implementation("org.assertj:assertj-core:latest.release")
                 implementation("org.junit.platform:junit-platform-suite-api")

rewrite-python/src/integTest/java/org/openrewrite/python/ParseProjectIntegTest.java

@@ -22,6 +22,7 @@
 import org.junit.jupiter.api.io.TempDir;
 import org.openrewrite.InMemoryExecutionContext;
 import org.openrewrite.SourceFile;
+import org.openrewrite.json.tree.Json;
 import org.openrewrite.python.marker.PythonResolutionResult;
 import org.openrewrite.python.rpc.PythonRewriteRpc;
 import org.openrewrite.text.PlainText;
@@ -225,6 +226,47 @@ void includesRequirementsTxt() throws IOException {
         assertThat(reqsTxt.getMarkers().findFirst(PythonResolutionResult.class)).isPresent();
     }
 
+    @Test
+    @Timeout(value = 60, unit = TimeUnit.SECONDS)
+    void includesPipfile() throws IOException {
+        Path projectDir = tempDir.resolve("with_pipfile");
+        Files.createDirectories(projectDir);
+
+        Files.writeString(projectDir.resolve("main.py"), "x = 1");
+        Files.writeString(projectDir.resolve("Pipfile"), """
+                [packages]
+                requests = ">=2.28.0"
+                """);
+        Files.writeString(projectDir.resolve("Pipfile.lock"), """
+                {
+                    "_meta": {"sources": [{"url": "https://pypi.org/simple"}]},
+                    "default": {"requests": {"version": "==2.31.0"}},
+                    "develop": {}
+                }
+                """);
+
+        List<SourceFile> sources = client()
+                .parseProject(projectDir, new InMemoryExecutionContext())
+                .collect(Collectors.toList());
+
+        assertThat(sources)
+                .extracting(sf -> sf.getSourcePath().getFileName().toString())
+                .contains("main.py", "Pipfile", "Pipfile.lock");
+
+        SourceFile pipfile = sources.stream()
+                .filter(sf -> sf.getSourcePath().getFileName().toString().equals("Pipfile"))
+                .findFirst()
+                .orElseThrow();
+        assertThat(pipfile).isInstanceOf(Toml.Document.class);
+        assertThat(pipfile.getMarkers().findFirst(PythonResolutionResult.class)).isPresent();
+
+        SourceFile pipfileLock = sources.stream()
+                .filter(sf -> sf.getSourcePath().getFileName().toString().equals("Pipfile.lock"))
+                .findFirst()
+                .orElseThrow();
+        assertThat(pipfileLock).isInstanceOf(Json.Document.class);
+    }
+
     @Test
     @Timeout(value = 60, unit = TimeUnit.SECONDS)
     void pyprojectTomlTakesPriorityOverRequirementsTxt() throws IOException {

rewrite-python/src/main/java/org/openrewrite/python/AddDependency.java

@@ -19,21 +19,24 @@
 import lombok.Value;
 import org.jspecify.annotations.Nullable;
 import org.openrewrite.*;
+import org.openrewrite.json.tree.Json;
+import org.openrewrite.marker.Markup;
+import org.openrewrite.python.internal.LockFileRegeneration;
 import org.openrewrite.python.internal.PyProjectHelper;
-import org.openrewrite.python.internal.PythonDependencyExecutionContextView;
 import org.openrewrite.python.trait.PythonDependencyFile;
 import org.openrewrite.toml.tree.Toml;
 
 import java.nio.file.Path;
 import java.util.Collections;
-import java.util.HashSet;
+import java.util.HashMap;
 import java.util.Map;
-import java.util.Set;
+import java.util.function.Function;
 
 /**
  * Add a dependency to a Python project. Supports {@code pyproject.toml}
  * (with scope and group targeting), {@code requirements.txt}, and {@code Pipfile}.
- * When uv is available, the uv.lock file is regenerated to reflect the change.
+ * When the matching package manager is available on {@code PATH}, the lock file
+ * (uv.lock for pyproject, Pipfile.lock for Pipfile) is regenerated to reflect the change.
  */
 @EqualsAndHashCode(callSuper = false)
 @Value
@@ -90,11 +93,22 @@ public String getInstanceNameSuffix() {
     public String getDescription() {
         return "Add a dependency to a Python project. Supports `pyproject.toml` " +
                 "(with scope/group targeting), `requirements.txt`, and `Pipfile`. " +
-                "When `uv` is available, the `uv.lock` file is regenerated.";
+                "When the matching package manager (`uv` or `pipenv`) is available, " +
+                "the corresponding lock file (`uv.lock` or `Pipfile.lock`) is regenerated. " +
+                "Not safe to use as a precondition: invokes the package manager and " +
+                "publishes per-project state shared with other dependency recipes.";
     }
 
     static class Accumulator {
-        final Set<Path> projectsToUpdate = new HashSet<>();
+        final Map<Path, ProjectState> projects = new HashMap<>();
+        final Map<Path, Path> lockToDeps = new HashMap<>();
+    }
+
+    static class ProjectState {
+        @Nullable SourceFile capturedDepsFile;
+        @Nullable String capturedLockContent;
+        @Nullable SourceFile modifiedDepsFile;
+        LockFileRegeneration.@Nullable Result regenResult;
     }
 
     @Override
@@ -114,26 +128,40 @@ public Tree preVisit(Tree tree, ExecutionContext ctx) {
                     return tree;
                 }
                 SourceFile sourceFile = (SourceFile) tree;
-                if (tree instanceof Toml.Document && sourceFile.getSourcePath().endsWith("uv.lock")) {
-                    PythonDependencyExecutionContextView.view(ctx).getExistingLockContents().put(
-                            PyProjectHelper.correspondingPyprojectPath(sourceFile.getSourcePath()),
-                            ((Toml.Document) tree).printAll());
+                Path sourcePath = sourceFile.getSourcePath();
+
+                if (tree instanceof Toml.Document && sourcePath.endsWith("uv.lock")) {
+                    Path depsPath = PyProjectHelper.correspondingPyprojectPath(sourcePath);
+                    ProjectState ps = acc.projects.computeIfAbsent(depsPath, k -> new ProjectState());
+                    ps.capturedLockContent = ((Toml.Document) tree).printAll();
+                    acc.lockToDeps.put(sourcePath, depsPath);
+                    return tree;
+                }
+                if (tree instanceof Json.Document && sourcePath.endsWith("Pipfile.lock")) {
+                    Path depsPath = PyProjectHelper.correspondingPipfilePath(sourcePath);
+                    ProjectState ps = acc.projects.computeIfAbsent(depsPath, k -> new ProjectState());
+                    ps.capturedLockContent = ((Json.Document) tree).printAll();
+                    acc.lockToDeps.put(sourcePath, depsPath);
                     return tree;
                 }
+
                 PythonDependencyFile trait = matcher.get(getCursor()).orElse(null);
-                if (trait != null && PyProjectHelper.findDependencyInScope(trait.getMarker(), packageName, scope, groupName) == null) {
-                    acc.projectsToUpdate.add(sourceFile.getSourcePath());
+                if (trait != null) {
+                    ProjectState ps = acc.projects.computeIfAbsent(sourcePath, k -> new ProjectState());
+                    ps.capturedDepsFile = sourceFile;
                 }
                 return tree;
             }
         };
     }
 
+    private boolean matchesAddDependency(PythonDependencyFile trait) {
+        return PyProjectHelper.findDependencyInScope(
+                trait.getMarker(), packageName, scope, groupName) == null;
+    }
+
     @Override
     public TreeVisitor<?, ExecutionContext> getVisitor(Accumulator acc) {
-        if (acc.projectsToUpdate.isEmpty()) {
-            return TreeVisitor.noop();
-        }
         return new TreeVisitor<Tree, ExecutionContext>() {
             final PythonDependencyFile.Matcher matcher = new PythonDependencyFile.Matcher();
 
@@ -146,27 +174,74 @@ public Tree preVisit(Tree tree, ExecutionContext ctx) {
                 SourceFile sourceFile = (SourceFile) tree;
                 Path sourcePath = sourceFile.getSourcePath();
 
-                if (acc.projectsToUpdate.contains(sourcePath)) {
+                ProjectState ps = acc.projects.get(sourcePath);
+                if (ps != null) {
                     PythonDependencyFile trait = matcher.get(getCursor()).orElse(null);
-                    if (trait != null) {
-                        String ver = version != null ? version : "";
-                        Map<String, String> additions = Collections.singletonMap(packageName, ver);
-                        PythonDependencyFile updated = trait.withAddedDependencies(additions, scope, groupName);
-                        if (updated.getTree() != tree) {
-                            return updated.afterModification(ctx);
+                    if (trait != null && matchesAddDependency(trait)) {
+                        ensureComputed(ps, trait);
+                    }
+                    if (ps.modifiedDepsFile != null) {
+                        SourceFile out = ps.modifiedDepsFile;
+                        if (ps.regenResult != null && !ps.regenResult.isSuccess()) {
+                            out = Markup.warn(out, new RuntimeException(
+                                    "lock regeneration failed: " + ps.regenResult.getErrorMessage()));
                         }
+                        PyProjectHelper.putLiveDepsTree(ctx, sourcePath, out);
+                        return out;
                     }
                 }
 
-                if (tree instanceof Toml.Document && sourcePath.endsWith("uv.lock")) {
-                    Toml.Document updatedLock = PyProjectHelper.maybeUpdateUvLock((Toml.Document) tree, ctx);
-                    if (updatedLock != null) {
-                        return updatedLock;
+                Path depsPath = acc.lockToDeps.get(sourcePath);
+                if (depsPath == null) {
+                    return tree;
+                }
+                ProjectState lockPs = acc.projects.get(depsPath);
+                if (lockPs == null) {
+                    return tree;
+                }
+                if (lockPs.modifiedDepsFile == null) {
+                    SourceFile depsTree = PyProjectHelper.getLiveDepsTree(ctx, depsPath);
+                    if (depsTree == null) {
+                        depsTree = lockPs.capturedDepsFile;
+                    }
+                    if (depsTree != null) {
+                        Cursor synth = new Cursor(new Cursor(null, Cursor.ROOT_VALUE), depsTree);
+                        PythonDependencyFile trait = matcher.get(synth).orElse(null);
+                        if (trait != null && matchesAddDependency(trait)) {
+                            ensureComputed(lockPs, trait);
+                            if (lockPs.modifiedDepsFile != null) {
+                                PyProjectHelper.putLiveDepsTree(ctx, depsPath, lockPs.modifiedDepsFile);
+                            }
+                        }
+                    }
+                }
+                if (lockPs.regenResult != null && lockPs.regenResult.isSuccess()) {
+                    String lockContent = lockPs.regenResult.getLockFileContent();
+                    if (tree instanceof Toml.Document) {
+                        return PyProjectHelper.reparseToml((Toml.Document) tree, lockContent);
+                    }
+                    if (tree instanceof Json.Document) {
+                        return PyProjectHelper.reparseJson((Json.Document) tree, lockContent);
                     }
                 }
-
                 return tree;
             }
+
+            private void ensureComputed(ProjectState ps, PythonDependencyFile trait) {
+                if (ps.modifiedDepsFile != null) {
+                    return;
+                }
+                String ver = version != null ? version : "";
+                Map<String, String> additions = Collections.singletonMap(packageName, ver);
+                Function<PythonDependencyFile, PythonDependencyFile> editFn =
+                        t -> t.withAddedDependencies(additions, scope, groupName);
+                PyProjectHelper.EditAndRegenerateResult r =
+                        PyProjectHelper.editAndRegenerate(trait, editFn, ps.capturedLockContent);
+                if (r.isChanged()) {
+                    ps.modifiedDepsFile = r.getModifiedDepsFile();
+                    ps.regenResult = r.getRegenResult();
+                }
+            }
         };
     }