From 304953657f583da35b1d7d2d2d6799d3e16583ed Mon Sep 17 00:00:00 2001 From: Christopher Dilks Date: Tue, 21 Apr 2026 13:53:29 -0400 Subject: [PATCH 1/7] feat: RG-K sp24 notes --- qadb/notes/rgk_sp24.md | 43 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 qadb/notes/rgk_sp24.md diff --git a/qadb/notes/rgk_sp24.md b/qadb/notes/rgk_sp24.md new file mode 100644 index 000000000..f76f6b38b --- /dev/null +++ b/qadb/notes/rgk_sp24.md @@ -0,0 +1,43 @@ +# Run Group K, Spring 2024, Pass 1 + +We will use SIDIS skims from two datasets: one for a 6.4 GeV beam, and another for a 8.5 GeV beam; we will keep them separate, to be consistent with what we did for Fall 2018 RG-K data: +- `rgk_sp24_6.4GeV`: from `/cache/clas12/rg-k/production/recon/spring2024/pass1/6395MeV/dst/train/skim1` +- `rgk_sp24_8.5GeV`: from `/cache/clas12/rg-k/production/recon/spring2024/pass1/8477MeV/dst/train/skim1` + +Before anything, cross check the train and DST run lists: +```bash +bin/qtl xtrain /mss/clas12/rg-k/production/recon/spring2024/pass1/6395MeV/dst/train/skim1 /mss/clas12/rg-k/production/recon/spring2024/pass1/6395MeV/dst/recon +bin/qtl xtrain /mss/clas12/rg-k/production/recon/spring2024/pass1/8477MeV/dst/train/skim1 /mss/clas12/rg-k/production/recon/spring2024/pass1/8477MeV/dst/recon +``` + +## Reheat + +**TODO**: +- [ ] check if needed; see Fall 2018 RG-K notes +- [ ] if reheating is needed, fix `qtl histogram` paths below + +## Run monitoring + +> [!IMPORTANT] +> Check any run-dependent settings in `qa-physics/monitorRead.groovy`, such as beam energy. + +```bash +qtl histogram -d rgk_sp24_6.4GeV --flatdir --focus-physics /cache/clas12/rg-k/production/recon/spring2024/pass1/6395MeV/dst/train/skim1 +qtl histogram -d rgk_sp24_8.5GeV --flatdir --focus-physics /cache/clas12/rg-k/production/recon/spring2024/pass1/8477MeV/dst/train/skim1 +``` + +## Double check that we have all the runs + +> [!IMPORTANT] +> In case any runs disappeared from `/cache` while running monitoring or reheating, be sure to cross check the output runs with those from `/mss` + +## Make timelines + +Make the timelines, separately for each of the 2 datasets: +```bash +bin/qtl physics -d rgk_sp24_6.4GeV -p rgk/pass2/qa/sp24_6.4GeV +bin/qtl physics -d rgk_sp24_8.5GeV -p rgk/pass2/qa/sp24_8.5GeV +``` + +> [!WARNING] +> You may not have permission to publish to this directory; if not, publish to another place and ask the chef to copy the files From 4cd3edb17455a8e526ca668da5ad0606418b0fd8 Mon Sep 17 00:00:00 2001 From: Christopher Dilks Date: Tue, 21 Apr 2026 15:06:18 -0400 Subject: [PATCH 2/7] feat: RG-E notes --- qadb/notes/rge_sp24.md | 49 ++++++++++++++++++++++++++++++++++++++++++ qadb/notes/rgk_sp24.md | 11 ++++++++-- 2 files changed, 58 insertions(+), 2 deletions(-) create mode 100644 qadb/notes/rge_sp24.md diff --git a/qadb/notes/rge_sp24.md b/qadb/notes/rge_sp24.md new file mode 100644 index 000000000..3bfe7a14d --- /dev/null +++ b/qadb/notes/rge_sp24.md @@ -0,0 +1,49 @@ +# Run Group E, Spring 2024, Pass 1 + +There is no SIDIS-like train, so let's use DST files for the QA. + +Similarly to RG-C, there are several targets that we'll need to combine; let's also combine torus polarities. + +The DSTs are found in: +``` +/cache/clas12/rg-e/production/spring2024/pass1/*/*/dst/recon +``` + +## Reheat + +**TODO**: +- [ ] check if needed; see Fall 2018 RG-K notes +- [ ] if reheating is needed, fix `qtl histogram` paths below + +## Run monitoring + +> [!IMPORTANT] +> Check any run-dependent settings in `qa-physics/monitorRead.groovy`, such as beam energy. + +check cache: +```bash +bin/qtl histogram -d rge_sp24 --check-cache --flatdir --focus-physics $(ls -d /cache/clas12/rg-e/production/spring2024/pass1/*/*/dst/recon) +``` + +run: +```bash +bin/qtl histogram -d rge_sp24 --flatdir --focus-physics $(ls -d /cache/clas12/rg-e/production/spring2024/pass1/*/*/dst/recon) +``` + +## Double check that we have all the runs + +> [!IMPORTANT] +> In case any runs disappeared from `/cache` while running monitoring or reheating, be sure to cross check the output runs with those from `/mss` + +## Make timelines + +**TODO**: +- [ ] populate `/data/metadata` for RG-E's target types + +Make the timelines: +```bash +bin/qtl physics -d rge_sp24 -p rgk/qa_pass1_sp24 +``` + +> [!WARNING] +> You may not have permission to publish to this directory; if not, publish to another place and ask the chef to copy the files diff --git a/qadb/notes/rgk_sp24.md b/qadb/notes/rgk_sp24.md index f76f6b38b..0f0d3a8e4 100644 --- a/qadb/notes/rgk_sp24.md +++ b/qadb/notes/rgk_sp24.md @@ -21,9 +21,16 @@ bin/qtl xtrain /mss/clas12/rg-k/production/recon/spring2024/pass1/8477MeV/dst/tr > [!IMPORTANT] > Check any run-dependent settings in `qa-physics/monitorRead.groovy`, such as beam energy. +check cache: ```bash -qtl histogram -d rgk_sp24_6.4GeV --flatdir --focus-physics /cache/clas12/rg-k/production/recon/spring2024/pass1/6395MeV/dst/train/skim1 -qtl histogram -d rgk_sp24_8.5GeV --flatdir --focus-physics /cache/clas12/rg-k/production/recon/spring2024/pass1/8477MeV/dst/train/skim1 +bin/qtl histogram -d rgk_sp24_6.4GeV --check-cache --flatdir --focus-physics /cache/clas12/rg-k/production/recon/spring2024/pass1/6395MeV/dst/train/skim1 +bin/qtl histogram -d rgk_sp24_8.5GeV --check-cache --flatdir --focus-physics /cache/clas12/rg-k/production/recon/spring2024/pass1/8477MeV/dst/train/skim1 +``` + +run: +```bash +bin/qtl histogram -d rgk_sp24_6.4GeV --flatdir --focus-physics /cache/clas12/rg-k/production/recon/spring2024/pass1/6395MeV/dst/train/skim1 +bin/qtl histogram -d rgk_sp24_8.5GeV --flatdir --focus-physics /cache/clas12/rg-k/production/recon/spring2024/pass1/8477MeV/dst/train/skim1 ``` ## Double check that we have all the runs From 8573bf6d0f1328ea079a1b3150acad947d1e8200 Mon Sep 17 00:00:00 2001 From: Christopher Dilks Date: Tue, 21 Apr 2026 15:27:40 -0400 Subject: [PATCH 3/7] fix: run number parsing --- bin/qtl-histogram | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/bin/qtl-histogram b/bin/qtl-histogram index 7a3b8d096..0556e1c05 100755 --- a/bin/qtl-histogram +++ b/bin/qtl-histogram @@ -300,7 +300,7 @@ declare -A runnumHash # `rdirs` element -> run number for rdir in ${rdirs[@]}; do # get the run number, either from `rdir` basename (fast), or from `RUN::config` (slow) [[ ! -e $rdir ]] && printError "the run file/directory '$rdir' does not exist" && continue - runnum=$(basename $rdir | grep -m1 -o -E "[0-9]+" || echo '') # first, try from run directory (or file) basename + runnum=$(basename $rdir | grep -m1 -o -E "[0-9]+" | tail -n1 || echo '') # first, try from run directory (or file) basename if [ -z "$runnum" ] || ${modes['swifjob']}; then # otherwise, use RUN::config from a HIPO file (NOTE: assumes all HIPO files have the same run number) if ${modes['flatdir']}; then $TIMELINESRC/libexec/hipo-check.sh $rdir @@ -313,7 +313,8 @@ for rdir in ${rdirs[@]}; do runnum=$($TIMELINESRC/libexec/run-groovy-timeline.sh $TIMELINESRC/libexec/get-run-number.groovy $firstHipo | tail -n1 | grep -m1 -o -E "[0-9]+" || echo '') fi fi - [ -z "$runnum" -o $runnum -eq 0 ] && printError "unknown run number for '$rdir'; ignoring it!" && continue + [ -z "$runnum" ] && printError "unknown run number for '$rdir'; ignoring it!" && continue + [ $runnum -eq 0 ] && printError "unknown run number for '$rdir'; ignoring it!" && continue runnum=$((10#$runnum)) runnumHash[$rdir]=$runnum done From 3a9a0052081b52dec5cab40aa9809dd54425a457 Mon Sep 17 00:00:00 2001 From: Christopher Dilks Date: Tue, 21 Apr 2026 15:34:57 -0400 Subject: [PATCH 4/7] fix(doc): `--rundir` --- qadb/notes/rge_sp24.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/qadb/notes/rge_sp24.md b/qadb/notes/rge_sp24.md index 3bfe7a14d..690cdee72 100644 --- a/qadb/notes/rge_sp24.md +++ b/qadb/notes/rge_sp24.md @@ -20,14 +20,17 @@ The DSTs are found in: > [!IMPORTANT] > Check any run-dependent settings in `qa-physics/monitorRead.groovy`, such as beam energy. +> [!NOTE] +> Using `--rundir` here rather than `--flatdir`, since we're using a directory of DST files. + check cache: ```bash -bin/qtl histogram -d rge_sp24 --check-cache --flatdir --focus-physics $(ls -d /cache/clas12/rg-e/production/spring2024/pass1/*/*/dst/recon) +bin/qtl histogram -d rge_sp24 --check-cache --rundir --focus-physics $(ls -d /cache/clas12/rg-e/production/spring2024/pass1/*/*/dst/recon) ``` run: ```bash -bin/qtl histogram -d rge_sp24 --flatdir --focus-physics $(ls -d /cache/clas12/rg-e/production/spring2024/pass1/*/*/dst/recon) +bin/qtl histogram -d rge_sp24 --rundir --focus-physics $(ls -d /cache/clas12/rg-e/production/spring2024/pass1/*/*/dst/recon) ``` ## Double check that we have all the runs From a151bce75e66aa8ff7b49c26a6df08195010fdab Mon Sep 17 00:00:00 2001 From: Christopher Dilks Date: Tue, 21 Apr 2026 18:58:41 -0400 Subject: [PATCH 5/7] feat: `--fast-ls` --- bin/qtl-histogram | 27 +++++++++++++++++++++++---- qadb/notes/rge_sp24.md | 6 +++--- 2 files changed, 26 insertions(+), 7 deletions(-) diff --git a/bin/qtl-histogram b/bin/qtl-histogram index 0556e1c05..dcbedbdcf 100755 --- a/bin/qtl-histogram +++ b/bin/qtl-histogram @@ -17,7 +17,7 @@ SLURM_LOG=/farm_out/%u/%x-%A_%a # default options dataset=test_v0 declare -A modes -for key in findhipo rundir eachdir flatdir single series submit check-cache swifjob focus-detectors focus-physics help; do +for key in findhipo rundir eachdir flatdir single series submit check-cache fast-ls swifjob focus-detectors focus-physics help; do modes[$key]=false done outputDir="" @@ -81,8 +81,9 @@ usageVerbose() { directory tree, e.g., runs grouped by target --rundir assume each specified [RUN_DIRECTORY] contains - subdirectories named as just run numbers; it is not - recommended to use wildcards for this option + subdirectories named as just run numbers; + - it is not recommended to use wildcards for this option + - see also \`--fast-ls\` option **this is the DEFAULT option** --eachdir assume each specified [RUN_DIRECTORY] is a single @@ -121,6 +122,13 @@ usageVerbose() { --focus-physics run histogramming for physics QA timelines + *** MISCELLANEOUS OPTIONS + + --fast-ls use \`jasmine list-files\` to get lists of files, rather + than commands such as \`find\` or \`ls\` + - useful for options such as \`--rundir\` for DST files + - WARNING: assumes you already tested \`--check-cache\` + $sep EXAMPLES: @@ -367,7 +375,18 @@ for rdir in ${rdirs[@]}; do realpath $rdir > $inputListFile else [[ "$(realpath $rdir)" =~ /mss/ ]] && swifPrefix="mss:" || swifPrefix="file:" - realpath $rdir/*.hipo | sed "s;^;$swifPrefix;" > $inputListFile + if ${modes['fast-ls']}; then + rdir_mss=$(realpath $rdir | sed 's;^.*/cache/;/mss/;') + rdir_cache=$(realpath $rdir) + # fast-ls mode will use `jasmine list-files`, but requires the stub files on `mss` to exist; if a cook is ongoing, + # it's possible the `/mss` directory won't be populated yet, although files are already on `/cache`; for that case, + # fall back to "slow" ls (i.e., the command that runs when not using `--fast-ls`) + jasmine list-files -display json -under $rdir_mss | jq -r '.[].mss' | xargs -n1 basename | sed "s;^;${swifPrefix}${rdir_cache}/;" > $inputListFile \ + || { printWarning "falling back to slow ls..."; realpath $rdir/*.hipo | sed "s;^;$swifPrefix;" > $inputListFile; } + else + echo " (use \`--fast-ls\` if this seems too slow)" + realpath $rdir/*.hipo | sed "s;^;$swifPrefix;" > $inputListFile + fi fi # generate job scripts diff --git a/qadb/notes/rge_sp24.md b/qadb/notes/rge_sp24.md index 690cdee72..a4263c21a 100644 --- a/qadb/notes/rge_sp24.md +++ b/qadb/notes/rge_sp24.md @@ -21,16 +21,16 @@ The DSTs are found in: > Check any run-dependent settings in `qa-physics/monitorRead.groovy`, such as beam energy. > [!NOTE] -> Using `--rundir` here rather than `--flatdir`, since we're using a directory of DST files. +> Using `--rundir` and `--fast-ls` here rather than `--flatdir`, since we're using a directory of DST files. check cache: ```bash -bin/qtl histogram -d rge_sp24 --check-cache --rundir --focus-physics $(ls -d /cache/clas12/rg-e/production/spring2024/pass1/*/*/dst/recon) +bin/qtl histogram -d rge_sp24 --check-cache --rundir --fast-ls --focus-physics $(ls -d /cache/clas12/rg-e/production/spring2024/pass1/*/*/dst/recon) ``` run: ```bash -bin/qtl histogram -d rge_sp24 --rundir --focus-physics $(ls -d /cache/clas12/rg-e/production/spring2024/pass1/*/*/dst/recon) +bin/qtl histogram -d rge_sp24 --rundir --fast-ls --focus-physics $(ls -d /cache/clas12/rg-e/production/spring2024/pass1/*/*/dst/recon) ``` ## Double check that we have all the runs From 5be83e5e8f3cb331de9879c7873d541c0e6cac16 Mon Sep 17 00:00:00 2001 From: Christopher Dilks Date: Fri, 24 Apr 2026 11:16:18 -0400 Subject: [PATCH 6/7] fix: typo --- qadb/notes/rge_sp24.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qadb/notes/rge_sp24.md b/qadb/notes/rge_sp24.md index a4263c21a..d442ba0b4 100644 --- a/qadb/notes/rge_sp24.md +++ b/qadb/notes/rge_sp24.md @@ -45,7 +45,7 @@ bin/qtl histogram -d rge_sp24 --rundir --fast-ls --focus-physics $(ls -d /cache/ Make the timelines: ```bash -bin/qtl physics -d rge_sp24 -p rgk/qa_pass1_sp24 +bin/qtl physics -d rge_sp24 -p rge/qa_pass1_sp24 ``` > [!WARNING] From d159b56a70d9d959b01d079c2bab6e6ff5144812 Mon Sep 17 00:00:00 2001 From: Christopher Dilks Date: Fri, 24 Apr 2026 11:20:35 -0400 Subject: [PATCH 7/7] doc: tune usage --- bin/qtl-histogram | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/bin/qtl-histogram b/bin/qtl-histogram index dcbedbdcf..785fa99f6 100755 --- a/bin/qtl-histogram +++ b/bin/qtl-histogram @@ -83,7 +83,7 @@ usageVerbose() { --rundir assume each specified [RUN_DIRECTORY] contains subdirectories named as just run numbers; - it is not recommended to use wildcards for this option - - see also \`--fast-ls\` option + - see also the \`--fast-ls\` option **this is the DEFAULT option** --eachdir assume each specified [RUN_DIRECTORY] is a single @@ -125,9 +125,11 @@ usageVerbose() { *** MISCELLANEOUS OPTIONS --fast-ls use \`jasmine list-files\` to get lists of files, rather - than commands such as \`find\` or \`ls\` + than commands such as \`find\` or \`ls\`, since \`jasmine\` + may be much faster than the latter commands for tape dirs. + - WARNING: assumes you already tested \`--check-cache\`, + since \`jasmine\` uses \`/mss\` stub files - useful for options such as \`--rundir\` for DST files - - WARNING: assumes you already tested \`--check-cache\` $sep