Multi-tenant hardening + e2e robustness fixes

End-to-end test verified: project goes from create → conda env clone →
Deep Research (Gemini) → title → citations → Dev Phase 4-step loop →
real claude code experimenter → LaTeX compile → 2-page PDF, all using
the project owner's own keys with no global fallback.

## Multi-tenant: remove ALL global config fallbacks

ARK is a product now: every user must supply their own keys. No more
silently using the lab admin's key when a user hasn't configured one.

* `deep_research.py`: strip `get_gemini_api_key()` to env-var only;
  delete `_global_config()` and `save_gemini_api_key()`.
* `telegram.py`: drop `_load_global()`, `save()`,
  `_global_config_path()`. `TelegramConfig` only reads per-project
  config + `ARK_TELEGRAM_*` env vars.
* `orchestrator.py:_get_bot_model`, `agents.py:_get_ark_model`,
  `cli.py:_get_configured_model`: never read `~/.ark/config.yaml`;
  use per-project config or hardcoded default. Drop the now-unused
  `get_config_dir` import in orchestrator.py and agents.py.
* `cli.py:_run_deep_research_for_project`: prompt user to export
  `GEMINI_API_KEY` instead of saving to global config.
* `cli.py:cmd_setup_bot`: global telegram setup is gone — point users
  at `ark setup-bot <project>` or `ARK_TELEGRAM_*` env vars.
* `webapp/jobs.py` + `slurm_template.sh` + `webapp/utils/verify.py`:
  drop the `ARK_NO_GLOBAL_CONFIG=1` workaround flag — no longer
  needed because the fallback is fully removed.

## Robustness fixes hit during real e2e

* `webapp/jobs.py:poll_local_job`: zombie reap. Use
  `os.waitpid(pid, WNOHANG)` instead of `os.kill(pid, 0)` so finished
  wrapper subprocesses get reaped instead of staying as zombies that
  the poller mistakes for "still running".
* `agents.py:run_agent`: empty-run detection no longer uses length or
  elapsed time as a quality signal. Length/duration are not correlated
  with correctness — a good title is 60 chars, a good yes/no is 3
  chars. Only flag empty when `returncode != 0` or output is literally
  empty. Initialise `stdout`/`stderr` BEFORE the try block so the
  TimeoutExpired path doesn't `NameError` on `stderr`. Re-capture
  stdout/stderr inside the timeout handler with a short timeout.
* `pipeline.py:_parse_title_from_agent_output`: new helper that
  strips `Title:` / `Generated title:` label lines so a model that
  prefixes its answer with a label doesn't end up with the label
  saved as the actual paper title.
* `pipeline.py:_plan_experiments`: page-aware experiment budget. A
  1-page workshop poster doesn't need 8 experiments; tell the planner
  to plan exactly N experiments based on `venue_pages` so the
  experimenter agent can actually finish in its time budget.
* `pipeline.py:_run_experiments` + `execution.py`: bump experimenter
  agent timeout from 1800s to 3600s — parameter sweeps need it, and
  the orchestrator already handles per-experiment results streaming.

## webapp dashboard log SSE no longer double-renders

`webapp/routes.py:api_stream_log`: track which file is being tailed
and how many lines have been sent FROM THAT FILE. On the very first
iteration, skip past the existing content (the client just fetched
it via `/log?lines=300`); on a real log-file rotation
(env_provision.log → local_*.out), reset and send the new file from
the start. Previously the SSE always restarted from line 0 every
time the connection opened, replaying everything the initial
`loadLog()` had already rendered.

Author: JihaoXin

ark/agents.py

@@ -64,7 +64,6 @@ def _fmt_tok(n: int) -> str:
         return f"{n / 1_000:.1f}k"
     return str(n)
 
-from ark.paths import get_config_dir
 from ark.ui import (
     ElapsedTimer, RateLimitCountdown, agent_styled, styled, Style, Icons,
 )
@@ -190,16 +189,13 @@ def _build_path_boundary(self) -> str:
         )
 
     def _get_ark_model(self) -> str | None:
-        """Read ARK model preference from .ark/config.yaml. Returns None to use CLI default."""
-        config_file = get_config_dir() / "config.yaml"
-        try:
-            if config_file.exists():
-                with open(config_file) as f:
-                    cfg = yaml.safe_load(f) or {}
-                return cfg.get("model")
-        except Exception:
-            pass
-        return None
+        """
+        Return the ARK model from the project config, or None to use CLI default.
+
+        No global config fallback — ARK is multi-tenant; per-project config
+        must declare its own model.
+        """
+        return self.config.get("model")
 
     def _kill_process_tree(self, pid: int):
         """Kill a process and all its descendants."""
@@ -522,6 +518,8 @@ def run_agent(self, agent_type: str, task: str, timeout: int = 1800,
 
                 timer.start()
                 result = ""
+                stdout = ""
+                stderr = ""
                 usage_record = None  # populated when claude returns parseable JSON
 
                 try:
@@ -582,7 +580,14 @@ def run_agent(self, agent_type: str, task: str, timeout: int = 1800,
                     process.kill()
                     timer.stop()
                     self.log(f"Agent {agent_type} timed out ({timeout}s)", "WARN")
-                    stdout, _ = process.communicate()
+                    # Capture whatever stdout/stderr is available so the empty-run
+                    # detection + downstream `stderr` references don't NameError.
+                    try:
+                        stdout, stderr = process.communicate(timeout=10)
+                    except Exception:
+                        stdout, stderr = "", ""
+                    stdout = stdout or ""
+                    stderr = stderr or ""
                     # JSON envelope is usually missing on timeout (truncated mid-stream).
                     # Try once; on failure fall back to raw text and let empty-run handle it.
                     if self.model == "claude":
@@ -599,10 +604,20 @@ def run_agent(self, agent_type: str, task: str, timeout: int = 1800,
                 timer.stop()
                 elapsed = int(time.time() - start_time)
 
-                # Empty-run detection with auto-retry
-                MIN_AGENT_TIME = 15
-                MIN_RESULT_LEN = 100
-                is_empty = elapsed < MIN_AGENT_TIME and len(result.strip()) < MIN_RESULT_LEN
+                # Empty-run detection with auto-retry.
+                # "Empty" means the agent didn't do its job — that's a property
+                # of the *outcome*, not the *length* or *speed* of the response.
+                # A good title is 60 chars; a good yes/no is 3 chars; a good
+                # "find this file" is one line. Length is not a quality signal.
+                #
+                # The only honest signals for "this run was broken":
+                #   - process exited non-zero (claude code crashed / errored)
+                #   - process produced literally no output
+                stripped = result.strip()
+                is_empty = (
+                    process.returncode != 0
+                    or not stripped
+                )
                 if is_empty:
                     self.log(f"Agent [{agent_type}] empty-run detected (attempt {attempt}/{MAX_RETRIES}): ran only {elapsed}s, output only {len(result.strip())} chars", "WARN")
                     self.log(f"  returncode: {process.returncode}", "WARN")

ark/cli.py

@@ -43,15 +43,13 @@ def get_projects_dir() -> Path:
 
 
 def _get_configured_model(default: str = "claude-sonnet-4-6") -> str:
-    """Read model from .ark/config.yaml, falling back to default."""
-    config_file = get_config_dir() / "config.yaml"
-    try:
-        if config_file.exists():
-            with open(config_file) as f:
-                cfg = yaml.safe_load(f) or {}
-            return cfg.get("model", default)
-    except Exception:
-        pass
+    """
+    Return the default ARK model.
+
+    No global config fallback — ARK is multi-tenant; per-project config.yaml
+    must declare its own model. Callers needing a project-specific model
+    should read it from the project's own config.yaml.
+    """
     return default
 
 
@@ -1503,9 +1501,8 @@ def _cmd_new_wizard(args, name: str, project_dir: Path, pdf_spec: dict):
         # Check for Gemini API key
         from ark.deep_research import get_gemini_api_key
         if not get_gemini_api_key():
-            print(f"  {_c('Note:', Colors.YELLOW)} No Gemini API key found.")
-            print(f"  Set it: ark config {name} --set gemini_api_key=YOUR_KEY")
-            print(f"  Or add to ~/.ark/config.yaml")
+            print(f"  {_c('Note:', Colors.YELLOW)} No Gemini API key found in environment.")
+            print(f"  Export it before running: export GEMINI_API_KEY=YOUR_KEY")
         else:
             print(f"  {_c('✓', Colors.GREEN)} Gemini API key found")
 
@@ -1705,28 +1702,18 @@ def _finalize_project(name: str, project_dir: Path, config: dict,
 # ============================================================
 
 def _run_deep_research_for_project(config: dict, state_dir: Path, custom_query: str = None):
-    """Run Gemini Deep Research for a project. Auto-runs if API key is available."""
-    from ark.deep_research import (
-        run_deep_research,
-        get_gemini_api_key,
-        save_gemini_api_key,
-    )
+    """Run Gemini Deep Research for a project if GEMINI_API_KEY is set."""
+    from ark.deep_research import run_deep_research, get_gemini_api_key
 
     print()
     print(f"{_c('Deep Research (Gemini)', Colors.BOLD)}")
 
     api_key = get_gemini_api_key()
     if not api_key:
-        if sys.stdin.isatty():
-            print("  No Gemini API key found.")
-            api_key = prompt_input("  Enter Gemini API key").strip()
-            if api_key:
-                save_gemini_api_key(api_key)
-                print(f"  {_c('API key saved to ~/.ark/config.yaml', Colors.DIM)}")
-        if not api_key:
-            print(f"  {_c('Skipped: no API key. Set with: ark config gemini-key <KEY>', Colors.YELLOW)}")
-            print()
-            return None
+        print(f"  {_c('Skipped: no GEMINI_API_KEY in environment.', Colors.YELLOW)}")
+        print(f"  {_c('Export it before running: export GEMINI_API_KEY=YOUR_KEY', Colors.DIM)}")
+        print()
+        return None
 
     # Allow custom query interactively
     if custom_query is None and sys.stdin.isatty():
@@ -2816,85 +2803,16 @@ def cmd_setup_bot(args):
         _cmd_setup_bot_project(project)
         return
 
-    # Global setup
-    from ark.telegram import TelegramConfig
-
-    print()
-    print(f"  {_c('Telegram Setup (Global)', Colors.BOLD + Colors.CYAN)}")
-    print()
-
-    # Load existing global config
-    existing = TelegramConfig()
-    existing_token = existing._load_global().get("bot_token")  # global only, not project
-
-    if existing_token:
-        print(f"  {_c('Existing bot token found.', Colors.YELLOW)}")
-        if not prompt_yn("  Replace it?", default=False):
-            tg_token = existing_token
-        else:
-            tg_token = prompt_input("  New Bot Token").strip()
-    else:
-        print(f"  Get your Bot Token from @BotFather in Telegram (/newbot).")
-        tg_token = prompt_input("  Bot Token").strip()
-
-    if not tg_token:
-        print("Aborted.")
-        return
-
-    # Auto-detect chat_id via getUpdates
+    # Global telegram config is no longer supported. ARK is multi-tenant:
+    # each project must have its own bot token, configured per-project.
     print()
-    print(f"  {_c('Now go to Telegram and send any message to your bot.', Colors.BOLD)}")
-    input("  Press Enter when done... ")
-
-    import urllib.request, json as _json, time as _time
-    tg_chat_id = None
-    for attempt in range(3):
-        try:
-            url = f"https://api.telegram.org/bot{tg_token}/getUpdates"
-            with urllib.request.urlopen(url, timeout=10) as resp:
-                data = _json.loads(resp.read())
-            results = data.get("result", [])
-            if results:
-                last = results[-1]
-                msg = last.get("message") or last.get("edited_message") or {}
-                sender = msg.get("from", {})
-                tg_chat_id = str(sender.get("id") or msg.get("chat", {}).get("id", ""))
-                if tg_chat_id:
-                    print(f"  {_c(f'→ Chat ID detected: {tg_chat_id}', Colors.GREEN)}")
-                    break
-        except Exception as e:
-            print(f"  {_c(f'Warning: {e}', Colors.YELLOW)}")
-        if attempt < 2:
-            print(f"  No messages found yet, retrying in 3s...")
-            _time.sleep(3)
-
-    if not tg_chat_id:
-        print(f"  {_c('Could not auto-detect Chat ID.', Colors.YELLOW)}")
-        tg_chat_id = prompt_input("  Enter Chat ID manually").strip()
-
-    if not tg_chat_id:
-        print("Aborted.")
-        return
-
-    # Send test message
-    try:
-        url = f"https://api.telegram.org/bot{tg_token}/sendMessage"
-        data = _json.dumps({
-            "chat_id": tg_chat_id,
-            "text": "✅ ARK Telegram notifications configured!",
-            "parse_mode": "Markdown",
-        }).encode("utf-8")
-        req = urllib.request.Request(url, data=data, headers={"Content-Type": "application/json"})
-        urllib.request.urlopen(req, timeout=10)
-        print(f"  {_c('→ Test message sent! Check your Telegram.', Colors.GREEN)}")
-    except Exception as e:
-        print(f"  {_c(f'Warning: test message failed: {e}', Colors.YELLOW)}")
-
-    # Save to global config (~/.ark/telegram.yaml)
-    existing.save(tg_token, tg_chat_id)
-
+    print(f"  {_c('Global telegram setup is no longer supported.', Colors.YELLOW)}")
     print()
-    print(f"  {_c('Saved to ~/.ark/telegram.yaml (shared by all projects).', Colors.GREEN)}")
+    print(f"  ARK is multi-tenant — each project must have its own bot.")
+    print(f"  Configure per-project:    {_c('ark setup-bot <project_name>', Colors.BOLD)}")
+    print(f"  Or set env vars before running:")
+    print(f"    {_c('export ARK_TELEGRAM_BOT_TOKEN=...', Colors.DIM)}")
+    print(f"    {_c('export ARK_TELEGRAM_CHAT_ID=...', Colors.DIM)}")
     print()
 
 

ark/deep_research.py

@@ -9,72 +9,21 @@
 import os
 import threading
 import time
-import yaml
 from datetime import datetime
 from pathlib import Path
 
 
-from ark.paths import get_config_dir
-
-
-def _global_config() -> Path:
-    return get_config_dir() / "config.yaml"
-
-
 def get_gemini_api_key() -> str:
     """
-    Get Gemini API key from env var or global config.
+    Return the Gemini API key from the process environment.
 
-    When ``ARK_NO_GLOBAL_CONFIG=1`` is set (which the webapp does for every
-    orchestrator subprocess), the global ``.ark/config.yaml`` fallback is
-    skipped — only env vars are honored. This prevents one webapp user's
-    project from silently using another user's (or the lab admin's)
-    Gemini key when they haven't configured their own.
+    ARK is multi-tenant: each user must supply their own key. There is
+    no shared/global config fallback. The webapp injects the key from
+    the project owner's encrypted user record into the orchestrator
+    subprocess as an environment variable; CLI users must export
+    ``GEMINI_API_KEY`` (or the synonym ``GOOGLE_API_KEY``) themselves.
     """
-    # 1. Environment variable
-    key = os.environ.get("GEMINI_API_KEY") or os.environ.get("GOOGLE_API_KEY")
-    if key:
-        return key
-
-    # 2. Global config (skipped under webapp / multi-user mode)
-    no_global = os.environ.get("ARK_NO_GLOBAL_CONFIG", "").strip().lower()
-    if no_global and no_global not in ("0", "false", "no", "off"):
-        return ""
-
-    if _global_config().exists():
-        try:
-            with open(_global_config()) as f:
-                cfg = yaml.safe_load(f) or {}
-            key = cfg.get("gemini_api_key")
-            if key:
-                return key
-        except Exception:
-            pass
-
-    return ""
-
-
-def save_gemini_api_key(key: str):
-    """Save Gemini API key to global config."""
-    _global_config().parent.mkdir(parents=True, exist_ok=True)
-
-    cfg = {}
-    if _global_config().exists():
-        try:
-            with open(_global_config()) as f:
-                cfg = yaml.safe_load(f) or {}
-        except Exception:
-            pass
-
-    cfg["gemini_api_key"] = key
-    with open(_global_config(), "w") as f:
-        yaml.dump(cfg, f, default_flow_style=False)
-
-    # Restrict permissions
-    try:
-        os.chmod(_global_config(), 0o600)
-    except Exception:
-        pass
+    return os.environ.get("GEMINI_API_KEY") or os.environ.get("GOOGLE_API_KEY") or ""
 
 
 def build_research_query(config: dict) -> str:

ark/execution.py

@@ -552,7 +552,7 @@ def _run_experiment_task(self, issue: dict, action_plan: dict):
 2. Check whether result files were generated
 3. Evaluate whether the data supports the paper's arguments
 4. Update auto_research/state/findings.yaml with new findings
-""", timeout=1800)
+""", timeout=3600)
 
             # 3. Wait for jobs
             self.log_step("Waiting for experiment completion...", "info")

ark/orchestrator.py

@@ -32,7 +32,6 @@
 PROJECT_DIR = None
 
 from ark.memory import get_memory, SimpleMemory
-from ark.paths import get_config_dir
 from ark.agents import AgentMixin
 from ark.compiler import CompilerMixin
 from ark.execution import ExecutionMixin
@@ -362,16 +361,13 @@ def stop_telegram_listener(self):
         self.telegram.stop()
 
     def _get_bot_model(self) -> str:
-        """Read bot model preference from .ark/config.yaml."""
-        config_file = get_config_dir() / "config.yaml"
-        try:
-            if config_file.exists():
-                with open(config_file) as f:
-                    cfg = yaml.safe_load(f) or {}
-                return cfg.get("bot_model", "claude-sonnet-4-6")
-        except Exception:
-            pass
-        return "claude-sonnet-4-6"
+        """
+        Return the model used for Telegram bot replies.
+
+        Prefers a per-project ``bot_model`` from the project config; falls
+        back to the default. No global config fallback — ARK is multi-tenant.
+        """
+        return self.config.get("bot_model") or "claude-sonnet-4-6"
 
     def _handle_telegram_message(self, text: str):
         """Handle incoming Telegram message via Claude agent."""