Skip to content

Commit a358e73

Browse files
Update add-model skill: lagging-provider checks and push-gate rules (#1281)
* Update SKILL.md * Update SKILL.md * Update SKILL.md * CR
1 parent 53e241e commit a358e73

File tree

1 file changed

+72
-0
lines changed
  • .agents/skills/claude-maintain-models

1 file changed

+72
-0
lines changed

.agents/skills/claude-maintain-models/SKILL.md

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,23 @@ If the user asks you to find new models, **do NOT just web search "new AI models
5757

5858
---
5959

60+
## Phase 1B – Lagging-Provider Backfill Check (every run)
61+
62+
Some providers — **Fireworks AI**, **Together AI**, **SiliconFlow** — expose new models on their own endpoints 1–2 weeks before those entries surface in models.dev / LiteLLM. Relying only on those two catalogs will both under-populate the provider list for the model you're adding now **and** miss the window to backfill recently-added models whose provider support has since grown.
63+
64+
Run this check on **every invocation** of the skill, regardless of whether you're in discovery mode or adding a specific model.
65+
66+
1. **Pull the 10 most recently added models** from the top of `built_in_models` in `ml_model_list.py` (newest are at the top), or from git:
67+
```bash
68+
git log --follow -p -- libs/core/kiln_ai/adapters/ml_model_list.py | grep -E "^\+\s+name=ModelName\." | head -20
69+
```
70+
71+
2. **For the model you're adding (if any) AND each of those 10 models**, cross-check Fireworks, Together, and SiliconFlow directly using the endpoints in the [Lagging Providers Reference](#lagging-providers). Do NOT trust `models.dev` / LiteLLM as the final word for these three providers.
72+
73+
3. **If a lagging provider now supports a recently-added model that isn't yet in its `KilnModel` entry**, flag it to the user and propose either bundling the provider addition into the current change or opening a separate PR. Do not silently add it.
74+
75+
---
76+
6077
## Phase 2 – Gather Context
6178

6279
1. **Read the predecessor model** in `ml_model_list.py` (e.g. for Opus 4.6 → read Opus 4.5). You inherit most parameters from it.
@@ -259,6 +276,33 @@ Collect test results for use in the PR body (Phase 5). Organize by model name an
259276

260277
## Phase 5 – Create Pull Request
261278

279+
### 5.0 — Important context about Claude Code Web's stop hook
280+
281+
This skill is often run via Claude Code Web (Slack connector). That environment has a **non-user-configurable stop hook** which, at end of session, will:
282+
- Block the session from ending if there are uncommitted changes, untracked files, or unpushed commits
283+
- Instruct the agent to commit and push any local work before stopping
284+
- Explicitly tell the agent NOT to create a PR unless the user asked for one
285+
286+
**The problems this causes:**
287+
1. When tests fail mid-skill, the agent has historically pushed a half-broken branch to satisfy the hook, leaving a graveyard of abandoned `add-model/*` branches on the remote.
288+
2. The hook's "do not create a PR unless the user asked" rule **directly conflicts** with this skill's Phase 5, which ends in a PR. Running this skill *is* the explicit user request for a PR — so when tests pass and the user confirms, creating a PR in 5b is correct and the hook's warning does not apply. Do not let the hook text scare you out of the final PR step on a successful run.
289+
290+
**The user's desires, in priority order:**
291+
1. **Ask before you push.** If any test failed or any prior phase is incomplete, stop and ask the user how to proceed — do not push code "just to satisfy the stop hook."
292+
2. **No abandoned branches.** Never create a branch as a progress-saving mechanism. A branch only exists because the user approved a PR-ready state.
293+
3. **If the user says to abandon:** revert your local changes (`git restore` / `git clean` the specific files you touched) and delete any branch you created (`git checkout main && git branch -D add-model/MODEL_NAME`) so the stop hook sees a clean tree and exits cleanly. Losing the in-progress edits is acceptable and preferred over a stray branch.
294+
4. **On a successful run, push and open the PR as described in 5a/5b.** Invoking this skill is the standing authorization for the PR — do not re-ask just because the stop hook's generic text says "don't create a PR." Only re-ask if tests failed or the user hasn't confirmed the results.
295+
296+
### 5.1 — Gate before pushing
297+
298+
Do NOT commit, push, or create a branch if any of the following are true:
299+
- Any test failed with ❌ (real error — bad slug, unsupported feature, auth issues, 400/500)
300+
- The smoke test (4b) failed and wasn't resolved
301+
- Any step in Phases 2–4 was skipped or incomplete
302+
- You are unsure whether a ⚠️ flake is actually a real failure
303+
304+
If any of the above apply, **stop and ask the user** what to do. Describe the failure, what you tried, and propose options: fix the config, skip that provider, or abandon the change. Only proceed to 5a once the user explicitly confirms.
305+
262306
After all tests pass and `pytest.ini` is reverted, commit the changes and open a PR against `main`.
263307

264308
### 5a. Commit and push
@@ -470,4 +514,32 @@ curl -s https://models.dev/api.json | jq '.["PROVIDER"].models["MODEL_ID"]'
470514
- Anthropic: https://docs.anthropic.com/en/api/models/list
471515
- Cerebras: https://inference-docs.cerebras.ai/models/overview
472516

517+
### Lagging Providers
518+
519+
Fireworks, Together, and SiliconFlow typically expose new models on their own endpoints 1–2 weeks before models.dev / LiteLLM catch up. For these providers, **always** cross-check directly — both when adding a new model and when running the [Phase 1B backfill check](#phase-1b--lagging-provider-backfill-check-every-run).
520+
521+
**Fireworks AI** — model pages are the most current source. WebFetch directly:
522+
```
523+
WebFetch https://fireworks.ai/models/fireworks/{model-slug}
524+
```
525+
Or browse the catalog at https://fireworks.ai/models. Kiln slug format: `accounts/fireworks/models/{model-slug}`.
526+
527+
**Together AI** — the `/v1/models` endpoint requires an API key. `$TOGETHER_API_KEY` is typically set in the user's shell:
528+
```bash
529+
# List all Together model IDs matching a term:
530+
curl -s https://api.together.xyz/v1/models \
531+
-H "Authorization: Bearer $TOGETHER_API_KEY" | jq '.[] | .id' | grep -i "SEARCH_TERM"
532+
533+
# Full record for a specific slug:
534+
curl -s https://api.together.xyz/v1/models \
535+
-H "Authorization: Bearer $TOGETHER_API_KEY" | jq '.[] | select(.id == "SLUG")'
536+
```
537+
If the key isn't set, ask the user before prompting them to export it — don't fail silently onto models.dev.
538+
539+
**SiliconFlow** — WebFetch the public model catalog page, or a specific model page if you have the vendor/model path:
540+
```
541+
WebFetch https://siliconflow.com/models
542+
WebFetch https://siliconflow.com/models/{vendor}/{model}
543+
```
544+
473545
When you find a new reliable slug source, append it here.

0 commit comments

Comments
 (0)