Per-project conda env + multi-tenant isolation hardening

Each new project now provisions its own conda env at <project_dir>/.env
by cloning the configured base env (default: ark). Plus hardening to
ensure projects can't read each other's keys, packages, or state.

## Per-project conda env

* New status `initializing` while the env clones in the background.
  webapp returns immediately; orchestrator launch waits for env ready.
* Status surfaced in dashboard (badge, status line, all 3 i18n locales)
  and in Project Info ("Conda Env: <prefix path>").
* Background task sends Telegram updates: starting, env ready, failure.
* `provision_project_env` clones via `conda create --prefix --clone`,
  logs to `<pdir>/logs/env_provision.log`, idempotent on retry.
* `launch_local_job` and slurm template prefer `<pdir>/.env` via
  `--prefix`; legacy projects fall back to the shared env.
* `LocalBackend.conda_env` / `SlurmBackend.conda_env` point the agent
  prompt at the project-local env when present.
* Delete works unchanged: `shutil.rmtree(pdir)` removes the env too.

## Robust binary discovery (systemd PATH safe)

Add `find_conda_binary()` and `find_claude_binary()` that fall back
to `$CONDA_EXE`, common miniforge/anaconda paths, and `~/.nvm/.../bin`
when `shutil.which()` fails under systemd's bare PATH. Add
`build_subprocess_path()` to enrich orchestrator subprocess `PATH`
with claude bin, ~/.local/bin, and texlive 2025.

## Multi-tenant isolation

* `PYTHONNOUSERSITE=1` for orchestrator + verify subprocesses so no
  project can read packages from /home/<user>/.local cross-tenant.
* `ARK_NO_GLOBAL_CONFIG=1` env var (read by `deep_research.get_gemini_api_key`
  and `telegram.TelegramConfig._load_global`) blocks fallback to
  lab-wide configs when running under the webapp; each project must
  use only the keys passed in by its owning user.
* `verify_claude_cli` runs with isolated HOME, full PATH enrichment,
  and resolves the absolute claude binary path so settings-page key
  verification works under systemd.

## Escape hatch + config

* `ARK_FORCE_LOCAL=1` makes `slurm_available()` return False, used
  while the SLURM controller spool is full.
* New setting `PROJECT_BASE_CONDA_ENV` (default `ark`) controls which
  env new projects clone from. Empty disables provisioning entirely.

Author: JihaoXin

ark/compute.py

@@ -99,9 +99,14 @@ def job_prefix(self) -> str:
 
     @property
     def conda_env(self) -> str:
-        return (self._compute_config.get("conda_env")
-                or self.config.get("conda_env")
-                or self.project_name)
+        explicit = (self._compute_config.get("conda_env")
+                    or self.config.get("conda_env"))
+        if explicit:
+            return explicit
+        local_env = self.code_dir / ".env"
+        if (local_env / "conda-meta").is_dir():
+            return str(local_env)
+        return self.project_name
 
     @property
     def slurm_template(self) -> str:
@@ -202,9 +207,15 @@ class LocalBackend(ComputeBackend):
 
     @property
     def conda_env(self) -> str:
-        return (self._compute_config.get("conda_env")
-                or self.config.get("conda_env")
-                or self.project_name)
+        explicit = (self._compute_config.get("conda_env")
+                    or self.config.get("conda_env"))
+        if explicit:
+            return explicit
+        # Per-project env created by the webapp at <code_dir>/.env.
+        local_env = self.code_dir / ".env"
+        if (local_env / "conda-meta").is_dir():
+            return str(local_env)
+        return self.project_name
 
     @property
     def gpu_count(self) -> int:

ark/deep_research.py

@@ -22,13 +22,25 @@ def _global_config() -> Path:
 
 
 def get_gemini_api_key() -> str:
-    """Get Gemini API key from env var or global config."""
+    """
+    Get Gemini API key from env var or global config.
+
+    When ``ARK_NO_GLOBAL_CONFIG=1`` is set (which the webapp does for every
+    orchestrator subprocess), the global ``.ark/config.yaml`` fallback is
+    skipped — only env vars are honored. This prevents one webapp user's
+    project from silently using another user's (or the lab admin's)
+    Gemini key when they haven't configured their own.
+    """
     # 1. Environment variable
     key = os.environ.get("GEMINI_API_KEY") or os.environ.get("GOOGLE_API_KEY")
     if key:
         return key
 
-    # 2. Global config
+    # 2. Global config (skipped under webapp / multi-user mode)
+    no_global = os.environ.get("ARK_NO_GLOBAL_CONFIG", "").strip().lower()
+    if no_global and no_global not in ("0", "false", "no", "off"):
+        return ""
+
     if _global_config().exists():
         try:
             with open(_global_config()) as f:

ark/telegram.py

@@ -44,6 +44,13 @@ def __init__(self, project_config: dict = None):
 
     def _load_global(self) -> dict:
         if self._global is None:
+            # In webapp / multi-user mode, never read the lab-wide telegram
+            # config: each project must use its own bot/chat from the
+            # project record.
+            no_global = os.environ.get("ARK_NO_GLOBAL_CONFIG", "").strip().lower()
+            if no_global and no_global not in ("0", "false", "no", "off"):
+                self._global = {}
+                return self._global
             if self._global_config_path().exists():
                 try:
                     with open(self._global_config_path()) as f:

ark/webapp/config.py

@@ -28,6 +28,12 @@ def _env_file() -> Path:
     "SLURM_PARTITION": "",
     "SLURM_ACCOUNT": "",
     "SLURM_CONDA_ENV": "ark",
+    # Base conda env that each new project's per-project env is cloned from.
+    # Defaults to "ark" because that's the env that historically ran the
+    # orchestrator and has all the heavy deps (google-genai, numpy, pandas,
+    # matplotlib, weasyprint, pytest, …) that ark-dev/ark-prod intentionally
+    # omit. Set to empty to disable per-project env provisioning entirely.
+    "PROJECT_BASE_CONDA_ENV": "ark",
     "GOOGLE_CLIENT_ID": "",
     "GOOGLE_CLIENT_SECRET": "",
 }
@@ -121,6 +127,7 @@ def __init__(self):
         self.slurm_partition: str = merged.get("SLURM_PARTITION", "")
         self.slurm_account: str = merged.get("SLURM_ACCOUNT", "")
         self.slurm_conda_env: str = merged.get("SLURM_CONDA_ENV", "ark")
+        self.project_base_conda_env: str = merged.get("PROJECT_BASE_CONDA_ENV", "ark")
         self.slurm_gres: str = merged.get("SLURM_GRES", "")
         self.slurm_cpus_per_task: int = int(merged.get("SLURM_CPUS_PER_TASK", "4"))
         raw_domains = merged.get("EMAIL_DOMAINS", "")

ark/webapp/db.py

@@ -140,7 +140,7 @@ def get_all_projects(session: Session) -> list[Project]:
 
 def get_running_projects(session: Session) -> list[Project]:
     return list(session.exec(
-        select(Project).where(Project.status.in_(["queued", "running", "pending"]))
+        select(Project).where(Project.status.in_(["queued", "running", "pending", "initializing"]))
     ).all())
 
 

ark/webapp/jobs.py

@@ -16,12 +16,162 @@
 
 _SLURM_TEMPLATE = Path(__file__).parent / "slurm_template.sh"
 
+# Per-project conda env: lives at <project_dir>/.env as a `--prefix` env cloned
+# from the configured base env. Detected via the conda-meta directory which
+# every conda env has, even an empty one.
+PROJECT_ENV_DIRNAME = ".env"
+
+
+def project_env_prefix(project_dir: Path) -> Path:
+    return Path(project_dir) / PROJECT_ENV_DIRNAME
+
+
+def project_env_ready(project_dir: Path) -> bool:
+    return (project_env_prefix(project_dir) / "conda-meta").is_dir()
+
+
+def find_claude_binary() -> str | None:
+    """
+    Locate the ``claude`` CLI even when systemd's bare PATH doesn't include
+    the user's nvm/npm bin dir. Tries shutil.which, $HOME/.local/bin, then
+    every nvm node version's bin dir, then a few common npm prefixes.
+    """
+    found = shutil.which("claude")
+    if found:
+        return found
+    home = Path(os.path.expanduser("~"))
+    candidates: list[Path] = [home / ".local" / "bin" / "claude"]
+    nvm_dir = home / ".nvm" / "versions" / "node"
+    if nvm_dir.is_dir():
+        # Newest version first so we pick the actively used one.
+        try:
+            for v in sorted(nvm_dir.iterdir(), reverse=True):
+                candidates.append(v / "bin" / "claude")
+        except OSError:
+            pass
+    candidates += [
+        home / ".npm-global" / "bin" / "claude",
+        Path("/usr/local/bin/claude"),
+    ]
+    for c in candidates:
+        if c.is_file() and os.access(c, os.X_OK):
+            return str(c)
+    return None
+
+
+def build_subprocess_path(extra: list[str] | None = None) -> str:
+    """
+    Build a PATH string suitable for spawning ARK subprocesses (orchestrator,
+    claude CLI, etc.) when the parent process has a bare systemd PATH.
+    Prepends: claude binary dir, ~/.local/bin, texlive 2025 bin, plus any
+    caller-supplied dirs, then the existing PATH.
+    """
+    parts: list[str] = list(extra or [])
+    home = Path(os.path.expanduser("~"))
+
+    claude = find_claude_binary()
+    if claude:
+        parts.append(str(Path(claude).parent))
+
+    parts.append(str(home / ".local" / "bin"))
+
+    texlive = home / "texlive" / "2025" / "bin" / "x86_64-linux"
+    if texlive.is_dir():
+        parts.append(str(texlive))
+
+    existing = os.environ.get("PATH", "/usr/bin:/bin")
+    # Dedupe while preserving order.
+    seen: set[str] = set()
+    out: list[str] = []
+    for p in parts + existing.split(":"):
+        if p and p not in seen:
+            seen.add(p)
+            out.append(p)
+    return ":".join(out)
+
+
+def find_conda_binary() -> str | None:
+    """
+    Locate the conda binary even when PATH is bare (e.g. systemd unit with no
+    Environment=PATH=). Tries shutil.which, then $CONDA_EXE, then common
+    install prefixes under $HOME.
+    """
+    found = shutil.which("conda")
+    if found:
+        return found
+    env_var = os.environ.get("CONDA_EXE")
+    if env_var and Path(env_var).is_file():
+        return env_var
+    home = Path(os.path.expanduser("~"))
+    for candidate in (
+        home / "miniforge3" / "condabin" / "conda",
+        home / "miniforge3" / "bin" / "conda",
+        home / "miniconda3" / "condabin" / "conda",
+        home / "miniconda3" / "bin" / "conda",
+        home / "anaconda3" / "condabin" / "conda",
+        home / "anaconda3" / "bin" / "conda",
+        Path("/opt/conda/bin/conda"),
+    ):
+        if candidate.is_file():
+            return str(candidate)
+    return None
+
+
+def provision_project_env(project_dir: Path, base_env: str = "ark-dev",
+                          log_path: Path | None = None) -> tuple[bool, str]:
+    """
+    Create a per-project conda env at <project_dir>/.env by cloning ``base_env``.
+
+    Returns ``(success, message)``. Idempotent: returns success immediately if
+    the env is already present. Writes the conda command output to ``log_path``
+    (or <project_dir>/.env_provision.log) for debugging.
+    """
+    project_dir = Path(project_dir)
+    target = project_env_prefix(project_dir)
+    log_path = Path(log_path) if log_path else (project_dir / ".env_provision.log")
+    log_path.parent.mkdir(parents=True, exist_ok=True)
+
+    if project_env_ready(project_dir):
+        return True, f"already exists at {target}"
+
+    conda_bin = find_conda_binary()
+    if not conda_bin:
+        msg = ("conda binary not found (checked PATH, $CONDA_EXE, and common "
+               "miniforge/anaconda locations); cannot provision project env")
+        log_path.write_text(msg + "\n")
+        return False, msg
+
+    # Stale partial env from a prior failed clone — wipe before retrying.
+    if target.exists():
+        shutil.rmtree(target, ignore_errors=True)
+
+    cmd = [conda_bin, "create", "--prefix", str(target),
+           "--clone", base_env, "--yes"]
+    started = time.time()
+    try:
+        with open(log_path, "w") as lf:
+            lf.write(f"$ {' '.join(cmd)}\n")
+            lf.flush()
+            proc = subprocess.run(cmd, stdout=lf, stderr=subprocess.STDOUT)
+        elapsed = time.time() - started
+        if proc.returncode == 0 and project_env_ready(project_dir):
+            return True, f"cloned {base_env} in {elapsed:.1f}s"
+        return False, f"conda create failed (rc={proc.returncode}); see {log_path}"
+    except Exception as e:
+        return False, f"conda create raised {type(e).__name__}: {e}"
+
 
 def _run(cmd: list[str], **kwargs) -> subprocess.CompletedProcess:
     return subprocess.run(cmd, capture_output=True, text=True, **kwargs)
 
 
 def slurm_available() -> bool:
+    # Escape hatch: set ARK_FORCE_LOCAL=1 to bypass SLURM submission and run
+    # everything as local subprocesses (useful when slurmctld spool is full or
+    # the cluster is unavailable). Falsy values ("", "0", "false") are ignored.
+    force_local = os.environ.get("ARK_FORCE_LOCAL", "").strip().lower()
+    if force_local and force_local not in ("0", "false", "no", "off"):
+        return False
     return shutil.which("sbatch") is not None
 
 
@@ -202,11 +352,18 @@ def launch_local_job(
     exit_file = log_dir / "local_exit.txt"
     exit_file.unlink(missing_ok=True)
 
-    # Build the orchestrator command, preferring the configured conda env.
-    conda_env = getattr(settings, "slurm_conda_env", "") or ""
-    conda_bin = shutil.which("conda") if conda_env else None
-    if conda_bin and conda_env:
-        python_prefix = [conda_bin, "run", "--no-capture-output", "-n", conda_env, "python"]
+    # Build the orchestrator command, preferring the project-local conda env
+    # at <project_dir>/.env. Falls back to the named env from settings, then
+    # the webapp's own interpreter.
+    conda_bin = find_conda_binary()
+    local_env = project_env_prefix(project_dir)
+    fallback_env = getattr(settings, "slurm_conda_env", "") or ""
+    if conda_bin and project_env_ready(project_dir):
+        python_prefix = [conda_bin, "run", "--no-capture-output",
+                         "--prefix", str(local_env), "python"]
+    elif conda_bin and fallback_env:
+        python_prefix = [conda_bin, "run", "--no-capture-output",
+                         "-n", fallback_env, "python"]
     else:
         python_prefix = [sys.executable]
 
@@ -241,6 +398,20 @@ def launch_local_job(
 
     env["HOME"] = str(project_dir)
     env["XDG_CONFIG_HOME"] = str(project_dir / ".config")
+    # Disable Python's pip user-site discovery so projects are completely
+    # isolated — no project can read packages from /home/xinj/.local/... or
+    # any other user's user-site. The cloned per-project conda env is the
+    # ONLY source of Python packages for the orchestrator.
+    env["PYTHONNOUSERSITE"] = "1"
+    # Tell the orchestrator (and any ark.* code it loads) to NEVER fall back
+    # to lab-wide configs like /home/xinj/ARK/.ark/config.yaml. Each project
+    # must use only the api keys / oauth tokens that the webapp passed in
+    # via env vars from the project's owning user.
+    env["ARK_NO_GLOBAL_CONFIG"] = "1"
+    # Make sure the orchestrator's PATH can find the claude CLI (lives in
+    # ~/.nvm/.../bin), latexmk (~/.local/bin), and pdflatex (~/texlive/2025/...).
+    # systemd's bare PATH doesn't include any of these.
+    env["PATH"] = build_subprocess_path()
 
     with open(log_file, "w") as lf:
         proc = subprocess.Popen(