Skip to content

SKILL.md

Latest commit

 

History

History
250 lines (187 loc) · 10.6 KB

SKILL.md

File metadata and controls

250 lines (187 loc) · 10.6 KB
name macos-agent-ops
description Use Homebrew-installed macos-agent to run repeatable app automation checks and routine tasks on macOS.

macos-agent ops

Contract

Prereqs:

  • macOS host with Accessibility and Automation permissions granted for Terminal and target apps.
  • Homebrew-installed macos-agent available on PATH.
  • cliclick, osascript, and im-select available on PATH.
  • ABC/US input source enabled in macOS Input Sources.
  • For full AX surface (ax attr/action/session/watch), Hammerspoon runtime should be healthy (hs + hs.ipc).

Inputs:

  • Command entrypoint: $AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh.
  • Optional env vars:
    • MACOS_AGENT_OPS_INPUT_SOURCE_ID (preferred; example: abc) for deterministic input source target.
    • MACOS_AGENT_REAL_E2E_INPUT_SOURCE (legacy fallback) if MACOS_AGENT_OPS_INPUT_SOURCE_ID is unset.
    • MACOS_AGENT_OPS_SKIP_INPUT_SOURCE_SWITCH=1 to bypass automatic input source switch.
    • CODEX_MACOS_AGENT_AX_BACKEND=auto|hammerspoon|applescript to control AX backend preference.

Outputs:

  • macos-agent JSON/text command output.
  • Artifacts generated by macos-agent under $AGENT_HOME/out/ (screenshots, traces, debug bundles, step ledgers).

Exit codes:

  • 0: success
  • 1: runtime failure or missing runtime dependency
  • 2: usage error

Failure modes:

  • wait app-active timeouts because focus was stolen (Control Center/Spotlight/notifications).
  • window.activate failure when app relaunch/update state is stuck; wrapper now uses --reopen-on-fail.
  • missing im-select causes input-source failure before keyboard-input actions.
  • not authorized / Apple Events denial from TCC permissions.
  • AX target mismatch (selector too broad / stale node id) during ax operations.
  • AX gate failures (--gate-*) when app/window/selector readiness conditions are not met.
  • AX postcondition failures (--postcondition-*) when state does not change as expected after mutation.
  • observe screenshot --if-changed-baseline path missing or unreadable.
  • trace directory validation failures when using --trace/--trace-dir and path is not writable.

Scripts (only entrypoints)

  • $AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh

Workflow

  1. Resolve Homebrew macos-agent binary path:
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh where
  1. Ensure deterministic input source (default target: abc):
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh input-source
# or explicit source id
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh input-source --id com.apple.keylayout.ABC
  1. Run readiness checks (preflight + AX list smoke check):
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh doctor
# optional target for AX smoke
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh doctor --ax-app Arc
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh doctor --ax-bundle-id com.google.Chrome
  1. Inspect unified permission state directly (screen_recording / accessibility / automation / ready):
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json preflight --include-probes
  1. Run quick app foreground check with reopen recovery:
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh app-check --app Finder
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh app-check --app Arc --timeout-ms 15000
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh app-check --bundle-id com.google.Chrome
  1. Run AX-only health check (AX tree probe):
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh ax-check --app Arc --role AXWindow
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh ax-check --app Arc --role AXTextField --title-contains Search
  1. Use AX wait primitives before mutation to stabilize dynamic UI:
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json wait ax-present --app Arc --role AXButton --title-contains "Compose" \
  --match-strategy contains --timeout-ms 2500 --poll-ms 80

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json wait ax-unique --app Arc --role AXTextField --title-contains "Search" \
  --selector-explain --timeout-ms 2500 --poll-ms 80
  1. Run robust mutating AX actions with gating + postconditions:
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json ax click --app Arc --role AXButton --title-contains "Play" \
  --match-strategy contains --selector-explain --reselect-before-click \
  --fallback-order ax-press,ax-confirm,frame-center,coordinate \
  --gate-app-active --gate-window-present --gate-ax-unique \
  --postcondition-focused true --postcondition-timeout-ms 2000 \
  --allow-coordinate-fallback
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json ax type --app Arc --role AXTextField --title-contains "Search" \
  --text "lofi" --clear-first --submit --paste \
  --gate-app-active --gate-window-present --gate-ax-present \
  --postcondition-attribute AXValue --postcondition-attribute-value "lofi" \
  --postcondition-timeout-ms 2000 --allow-keyboard-fallback
  1. Use selector-frame screenshots for visual proof of target alignment:
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json observe screenshot --active-window \
  --role AXButton --title-contains "Play" --selector-padding 12 \
  --path "$AGENT_HOME/out/macos-agent-selector-$(date +%Y%m%d-%H%M%S).png"
  1. Use diff-aware screenshots to avoid noisy artifacts when nothing changed:
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json observe screenshot --active-window \
  --path "$AGENT_HOME/out/macos-agent-state.png" \
  --if-changed --if-changed-threshold 2
  1. Use one-shot debug bundle for triage artifacts:
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json debug bundle --active-window \
  --output-dir "$AGENT_HOME/out/macos-agent-debug-$(date +%Y%m%d-%H%M%S)"
  1. Run routine scripted scenarios:
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh scenario --file /path/to/scenario.json
  1. Pass through raw macos-agent commands with JSON errors + trace when needed:
$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json --error-format json --trace \
  input click --x 200 --y 160

Screenshot-Based Triage Rules

  • During runtime failure triage, the agent SHOULD capture an active-window screenshot before retry/remediation.
  • For wait app-active and window.activate failures, screenshot capture SHOULD be first-line diagnostics.
  • Screenshot artifacts MUST be written under $AGENT_HOME/out/ and SHOULD use timestamped filenames.
  • The agent SHOULD include screenshot path + failing command + error message in the final diagnostic summary.

Recommended command (via the skill entrypoint):

$AGENT_HOME/skills/tools/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json observe screenshot --active-window \
  --path "$AGENT_HOME/out/macos-agent-failure-$(date +%Y%m%d-%H%M%S).png"

Common Errors And Prevention

  • error: input source mismatch after switch attempt ...

    • Run .../macos-agent-ops.sh input-source --id abc and verify current is com.apple.keylayout.ABC or ...US.
    • Ensure im-select is installed: brew install im-select.
    • If you intentionally need non-ABC IME, set MACOS_AGENT_OPS_SKIP_INPUT_SOURCE_SWITCH=1.

  • error: timed out waiting for app-active ...

    • Keep hands off keyboard/mouse while checks are running.
    • Close Control Center/Spotlight overlays and reduce notification interruptions.
    • Increase timeout (--timeout-ms 12000 or higher) for slow app transitions.

  • error: ax.list ... timed out during wait ax-present / wait ax-unique

    • Run app-check first, then retry AX waits after app focus is stable.
    • Increase AX wait timeout/poll (--timeout-ms, --poll-ms) and narrow selectors.
    • Keep backend on auto unless you need to force Hammerspoon behavior (CODEX_MACOS_AGENT_AX_BACKEND).

  • window activate failed ...

    • Wrapper already uses --reopen-on-fail; rerun once to allow quit/relaunch recovery.
    • For flaky apps (e.g., Spotify updater), kill stale relauncher and retry.

  • not authorized / Apple Events failures

    • Re-run doctor and fix Accessibility/Automation in System Settings.

  • selector returned zero AX matches or ambiguous matches

    • Narrow selectors and use --match-strategy exact|prefix|suffix|regex when contains is too broad.
    • Use --selector-explain and wait ax-unique before ax click/ax type.
    • Prefer --reselect-before-click in dynamic pages where node ids churn.

  • gate failure (--gate-*) before mutation

    • Add/adjust readiness gates based on context (--gate-app-active, --gate-window-present, --gate-ax-present, --gate-ax-unique).
    • Increase gate timeout/poll for slower app transitions (--gate-timeout-ms, --gate-poll-ms).

  • postcondition failure (--postcondition-*) after mutation

    • Verify you are asserting the correct target/attribute (AXValue, focus state, etc.).
    • Increase postcondition timeout on animation-heavy apps (--postcondition-timeout-ms).

  • error: --if-changed-baseline path does not exist ...

    • Verify baseline path exists before running capture.
    • If baseline is optional, remove --if-changed-baseline and let current output file be baseline.

  • trace directory is not writable (trace.write errors)

    • Use a writable custom path with --trace-dir.
    • Prefer $AGENT_HOME/out/... for reproducible artifacts.

  • Homebrew macos-agent missing

    • Install and link with Homebrew: brew install macos-agent.

Practical Notes

  • Prefer canonical flags in examples:
    • --window-title-contains (instead of legacy --window-name)
    • input type --submit (instead of legacy --enter)
  • Keep --error-format json enabled in automation loops when you need machine-parseable failures.
  • Use debug bundle first when a run fails and root cause is unclear.
  • Parse preflight JSON permissions (screen_recording / accessibility / automation / ready) before mutating runs.

References

  • skills/tools/macos-agent-ops/references/e2e-ref-01-preflight-focus.md
  • skills/tools/macos-agent-ops/references/e2e-ref-02-finder-routine.md
  • skills/tools/macos-agent-ops/references/e2e-ref-03-matrix-routine.md
⚡