Skip to content

README.md

Latest commit

 

History

History
96 lines (83 loc) · 3.61 KB

README.md

File metadata and controls

96 lines (83 loc) · 3.61 KB

AiSOC hunts/ — hunt-as-code corpus

This directory holds AiSOC's hunt hypotheses as YAML. Each file describes a single threat hypothesis the agent layer can run on a schedule, scoring the target telemetry corpus against a set of expected indicators.

The corpus is part of the same eval-graded pipeline as detections and prompts: every hunt ships with a synthetic positive scenario in services/agents/tests/eval_data/synthetic_telemetry.jsonl and is scored by scripts/run_hunt_evals.py (or as part of scripts/run_evals.py when run in --all mode).

Schema

id: hunt-<slug>                # stable identifier, must be unique
name: Human readable name
description: |
  One-paragraph description of the hypothesis the hunt is testing.
version: 1.0.0
severity: low | medium | high | critical
category: identity | endpoint | cloud | network | data-exfil | application
tags:
  - mitre.attack.t1078         # MITRE technique IDs welcome
  - threat.<actor>             # optional threat actor hint
log_sources:                    # what telemetry the hunt expects
  - sysmon
  - m365_audit
  - linux_auditd
schedule:
  enabled: true
  interval_minutes: 60         # how often the scheduler should run this
  jitter_seconds: 60           # randomized jitter to avoid herd
hypothesis:
  question: |
    "Did any service account log into a workstation outside business hours
    and immediately enumerate domain groups?"
  indicators:
    - field: source
      equals: windows_security
    - field: EventID
      in: [4624, 4625]
    - field: TargetUserName
      regex: "^svc-.*"
expected:
  # synthetic positive scenario for eval grading
  positive_incident_id: INC-HUNT-001
  positive_template_id: hunt-svc-account-after-hours
  # at least this fraction of indicators must match for the hunt to fire
  min_match_score: 0.8
  # synthetic negative scenario id (a benign event sequence the hunt must NOT fire on)
  negative_incident_id: INC-HUNT-001-NEG
references:
  - https://attack.mitre.org/techniques/T1078/
author: AiSOC
created: '2026-05-05'
modified: '2026-05-05'

Eval grading

Each hunt is scored on:

  1. True positive rate — does the hunt fire on expected.positive_incident_id?
  2. False positive rate — does the hunt not fire on expected.negative_incident_id?
  3. Indicator quality — fraction of declared indicators that actually match the synthetic events (the substrate self-consistency check, like the rest of the eval harness).

The aggregate "Hunt Coverage" score is the macro-average of true positive rate across all enabled hunts. CI gates regression by ≥ 1pp, identical to the detection-as-code gate.

How the scheduler runs

services/agents runs an APScheduler AsyncIOScheduler job per enabled hunt. On each fire the hunt agent:

  1. Loads the hunt YAML from disk.
  2. Queries the telemetry warehouse (or the synthetic corpus in dev) for events matching hypothesis.indicators.
  3. If min_match_score is reached, persists a hunt finding as a first-class artifact alongside the Investigation Ledger (table hunt_findings).
  4. Writes a run record to hunt_runs regardless of whether the hunt fired.

Adding a hunt

  1. Drop a new YAML into hunts/.
  2. Add at least one synthetic positive scenario to services/agents/tests/eval_data/synthetic_telemetry.jsonl with the matching incident_id and template_id.
  3. Add a synthetic negative scenario (benign events that match some but not all indicators).
  4. Run python scripts/run_hunt_evals.py locally to confirm the hunt scores 100% true-positive and 0% false-positive.
  5. Open a PR — CI will re-run the hunt evals and gate on regression.
⚡