PipRecipeBundleResolver: pip install on the server side#7547
Merged
jkschneider merged 1 commit intomainfrom May 2, 2026
Merged
PipRecipeBundleResolver: pip install on the server side#7547jkschneider merged 1 commit intomainfrom
jkschneider merged 1 commit intomainfrom
Conversation
Mirrors the JS RPC server's design where `npm install` runs server-side during InstallRecipes. Today the Python RPC server's handle_install_recipes assumes the package is already importable; if it isn't, prepareRecipe later throws "Recipe not found". The JVM-side PipRecipeBundleResolver doesn't pip install either, so the contract was implicit: "some other process must have pre-installed this package", which is fragile and forced downstream callers to duplicate install logic. Fix on the Python side: - New `--recipe-install-dir` argv stores the install target globally. - `_pip_install_recipe_package` shells out to `python -m pip install --target <dir> <package>==<version>` and primes sys.path. - `handle_install_recipes`'s package-spec branch invokes it when the requested package isn't already importable at the requested version, before the existing import + activate flow. Fix on the Java side: - `PythonRewriteRpc` passes `--recipe-install-dir=<path>` to the Python process when the builder's `recipeInstallDir` is set. The existing PYTHONPATH wiring is unchanged, so a package that's already installed remains importable as before. The local-path branch (caller passes a directory containing an already- unpacked recipe package) is unchanged.
7d7d453 to
2dbbb53
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Aligns the pip recipe install flow with the npm one. Today:
JavaScriptRewriteRpc's server-sideInstallRecipeshandler shells out tonpm install <pkg>@<ver> --no-fund(rewrite-javascript/rewrite/src/rpc/request/install-recipes.ts:116). The JVM caller passes a package spec; the JS RPC server installs and activates.PythonRewriteRpc's server-sidehandle_install_recipesdoesn't install — its docstring explicitly says the package "should already be installed by the caller (e.g., via pip install --target)". The JVM-sidePipRecipeBundleResolver.resolve()callsinstallRecipes(packageName, version)without anypip install, so when the caller hasn't pre-installed (which is the common case),_import_and_activate_packagesilently fails to register the recipe and a laterprepareRecipethrowsRecipe not found: <id>.This PR makes the Python server install pip packages itself, matching the npm contract.
How
Python side (
rewrite-python/rewrite/src/rewrite/rpc/server.py)--recipe-install-dirargv parsed at startup; stored as_recipe_install_dir._pip_install_recipe_package(name, version, target_dir)shells out topython -m pip install --target <dir> <pkg>==<ver>, raises on non-zero exit, and primessys.path+importlib.invalidate_caches().handle_install_recipes's package-spec branch invokes it when the requested package isn't already importable at the requested version, before the existing import + activate flow. Local-path branch is unchanged.Java side (
PythonRewriteRpc.java)--recipe-install-dir=<path>is passed to the Python subprocess when the builder'srecipeInstallDiris set. The existing PYTHONPATH wiring stays so already-installed packages continue to import as before.Test plan
tests/rpc/test_server.py::test_handle_install_recipes_pip_installs_when_target_configured— locks the install command shape (-m pip install --target <dir> pkg==ver) and asserts the target dir lands onsys.path. Mockssubprocess.runso the test doesn't actually shell out to PyPI.:rewrite-python:compileJavapasses.tests/rpc/test_server.pycases still pass.Notes for downstream callers
If you use
PipRecipeBundleResolverand you weren't pre-installing recipe packages: setPythonRewriteRpc.builder().recipeInstallDir(<writable-dir>)and you're done — the server installs on demand. If you were pre-installing, no change is required:_is_package_installedshort-circuits and the existing path runs.