refactor!: top-level script "qtl" for running all timeline code

[c-dilks]

Summary

With this PR, chefs should just have to:

1. run the workflow with "qtl" model
2. run qtl analysis with (at least) options -i and -p; timelines will now be deployed automatically
   • ⚠️ this is a breaking change, since qtl analysis replaces run-detectors-timelines.sh AND deploy-timelines.sh

PR diff

Since some scripts were renamed and changed enough such that GitHub does not provide an easy-to-read diff, here are diffs of the main scripts:

run-monitoring.sh -> qtl-histogram

--- v2/bin/run-monitoring.sh	2025-04-28 10:35:31.100904313 -0400
+++ v3/bin/qtl-histogram	2025-04-28 10:35:44.023832013 -0400
@@ -1,526 +1,527 @@
 #!/usr/bin/env bash
 
 set -e
 set -u
 source $(dirname $0)/../libexec/environ.sh
 
 # constants ############################################################
-# max number of events for detector monitoring timelines
+# max number of events for detector histogramming timelines
 MAX_NUM_EVENTS=100000000
 # slurm settings
 SLURM_MEMORY=1600 # must be more than max heap size in $TIMELINE_JAVA_OPTS
 SLURM_TIME=10:00:00
 SLURM_LOG=/farm_out/%u/%x-%A_%a
 ########################################################################
 
 
 # default options
 dataset=test_v0
 declare -A modes
 for key in findhipo rundir eachdir flatdir single series submit check-cache swifjob focus-detectors focus-physics help; do
   modes[$key]=false
 done
 outputDir=""
 
 # usage
 sep="================================================================"
 usageTerse() {
   echo """
-  Run the timeline monitoring jobs
+  Run the timeline histogramming jobs.
+  NOTE: chefs typically run this program using the CLAS12 workflow
+
   $sep
-  USAGE: $0 -d [DATASET_NAME] [RUN_DIRECTORY]
+  USAGE: qtl histogram -d [DATASET_NAME] [RUN_DIRECTORY]
   $sep
 
      -d [DATASET_NAME]   unique dataset name, defined by the user, used for organization
 
      [RUN_DIRECTORY]     input data directory, with subdirectories organized by run,
                          **must be specified last**
 
   After running this script, run either or both of the suggested 'sbatch' command(s);
   one is for detector QA timelines and the other is for physics QA timelines.
   Output files will appear in ./outfiles/[DATASET_NAME]
 
-  For more options, run:
-    $0 --help
+  For more options, run with '--help'
   """ >&2
 }
 usageVerbose() {
   echo """
   $sep
-  USAGE: $0  [OPTIONS]...  [RUN_DIRECTORY]...
+  USAGE: qtl histogram  [OPTIONS]...  [RUN_DIRECTORY]...
   $sep
 
   REQUIRED ARGUMENTS:
 
     [RUN_DIRECTORY]...   One or more directories, each directory corresponds to
                          one run and should contain reconstructed HIPO files
                          - See \"INPUT FINDING OPTIONS\" below for more control,
                            so that you don't have to specify each run's directory
                          - A regexp or globbing (wildcards) can be used to
                            specify the list of directories as well, if your shell
                            supports it
                          - for a directory of files, with one run per file (e.g.,
                            SKIM files), use the option \`--flatdir\`
 
   $sep
 
   OPTIONS:
 
      -d [DATASET_NAME]      unique dataset name, defined by the user, used for organization
                             default = '$dataset'
 
      -o [OUTPUT_DIR]        custom output directory
                             default = ./outfiles/[DATASET_NAME]
 
      *** INPUT FINDING OPTIONS: control how the input HIPO files are found;
          choose only one:
 
        --findhipo     use \`find\` to find all HIPO files in each
                       [RUN_DIRECTORY]; this is useful if you have a
                       directory tree, e.g., runs grouped by target
 
        --rundir       assume each specified [RUN_DIRECTORY] contains
                       subdirectories named as just run numbers; it is not
                       recommended to use wildcards for this option
                       **this is the DEFAULT option**
 
        --eachdir      assume each specified [RUN_DIRECTORY] is a single
                       run's directory full of HIPO files (e.g., DST files)
 
        --flatdir      assume each specified [RUN_DIRECTORY] contains HIPO
                       files, with one run per file; use this option for SKIM files
 
        --check-cache  cross check /cache directories with tape stub directories
                       (/mss) and exit without creating or running any jobs; this is
                       useful if you are running QA on older DSTs which may no longer be
                       fully pinned on /cache
 
      *** EXECUTION CONTROL OPTIONS: choose only one, or the default will generate a
          Slurm job description and print out the suggested \`sbatch\` command
 
        --single    run only the first job, locally; useful for
                    testing before submitting jobs to slurm
 
        --series    run all jobs locally, one at a time; useful
                    for testing on systems without slurm
 
        --submit    submit the slurm jobs, rather than just
                    printing the \`sbatch\` command
 
        --swifjob   run this on a workflow runner, where the input
                    HIPO files are found in ./ and specifying [RUN_DIRECTORIES] is
                    not required; overrides some other settings; this is NOT meant
                    to be used interactively, but rather as a part of a workflow
 
      *** FOCUS OPTIONS: these options allow for running single types of jobs,
          rather than the default of running everything; you may specify more
          than one
 
-       --focus-detectors   run monitoring for detector QA timelines
+       --focus-detectors   run histogramming for detector QA timelines
 
-       --focus-physics     run monitoring for physics QA timelines
+       --focus-physics     run histogramming for physics QA timelines
 
   $sep
 
   EXAMPLES:
 
   $  $0 -v v1.0.0 --submit --rundir /volatile/mon
        -> submit slurm jobs for all numerical subdirectories of /volatile/mon/,
           where each subdirectory should be a run number; this is the most common usage
 
   $  $0 -v v1.0.0 --eachdir /volatile/mon/*
        -> generate the slurm script to run on all subdirectories of
           /volatile/mon/ no matter their name
 
   $  $0 -v v1.0.0 --single /volatile/mon/run*
        -> run on the first directory named run[RUNNUM], where [RUNNUM] is a run number
 
   """ # stream to stdout to permit grepping
 }
 if [ $# -lt 1 ]; then
   usageTerse
   exit 101
 fi
 
 # parse options
 while getopts "d:o:h-:" opt; do
   case $opt in
     d) dataset=$OPTARG;;
     o) outputDir=$OPTARG;;
     h) modes['help']=true;;
     -)
       for key in "${!modes[@]}"; do
         [ "$key" == "$OPTARG" ] && modes[$OPTARG]=true && break
       done
       [ -z "${modes[$OPTARG]-}" ] && printError "unknown option --$OPTARG" && exit 100
       ;;
     *) exit 100;;
   esac
 done
 shift $((OPTIND - 1))
 
 # print full usage guide
 if ${modes['help']}; then
   usageVerbose
   exit 101
 fi
 
 # set the input-finding method; use the DEFAULT one, if not set by user
 numTrueInputOpts=0
 for key in findhipo rundir eachdir flatdir; do
   if ${modes[$key]}; then
     numTrueInputOpts=$((numTrueInputOpts+1))
   fi
 done
 if [ $numTrueInputOpts -eq 0 ]; then
   modes['rundir']=true # set the DEFAULT option
 elif [ $numTrueInputOpts -gt 1 ]; then
   printError "more than one input-finding option set"
   exit 100
 fi
 
 # parse input directories
 rdirs=()
 if ${modes['swifjob']}; then
   rdirs=(.) # all input files reside in ./ on a workflow runner
 else
   [ $# == 0 ] && printError "no run directories specified" && exit 100
   rdirsArgs="$@"
   for topdir in ${rdirsArgs[@]}; do
     [[ "$topdir" =~ ^- ]] && printError "option '$topdir' must be specified before run directories" && exit 100
   done
   if ${modes['rundir']}; then
     for topdir in ${rdirsArgs[@]}; do
       if [ -d $topdir ]; then
         for subdir in $(ls $topdir | grep -E "[0-9]+"); do
           rdirs+=($(echo "$topdir/$subdir " | sed 's;//;/;g'))
         done
       else
         printError "run directory '$topdir' does not exist"
         exit 100
       fi
     done
   elif ${modes['eachdir']}; then
     rdirs=$@
   elif ${modes['flatdir']}; then
     rdirs=$@
   elif ${modes['findhipo']}; then
     for topdir in ${rdirsArgs[@]}; do
       echo "finding .hipo files in $topdir ....."
       fileList=$(find -L $topdir -type f -name "*.hipo")
       if [ -z "$fileList" ]; then
         printWarning "run directory '$topdir' has no HIPO files"
       else
         rdirs+=($(echo $fileList | xargs dirname | sort -u))
       fi
     done
   else
     printError "unknown input option"
     exit 100
   fi
 fi
 [ ${#rdirs[@]} -eq 0 ] && printError "no run directories found" && exit 100
 echo "done finding input run directories"
 
 # set and make output directory
 if ${modes['swifjob']}; then
   outputDir=$(pwd -P)/outfiles
 else
   [ -z "$outputDir" ] && outputDir=$(pwd -P)/outfiles/$dataset
 fi
 mkdir -p $outputDir
 
 # check focus options
 modes['focus-all']=true
 for key in focus-detectors focus-physics; do
   if ${modes[$key]}; then modes['focus-all']=false; fi
 done
 if ${modes['swifjob']} && ${modes['focus-all']}; then
   printError "option --swifjob must be used with either --focus-detectors or --focus-physics"
   exit 100
 fi
 
 # print arguments
 echo """
 Settings:
 $sep
 DATASET_NAME = $dataset
 OUTPUT_DIR   = $outputDir
 OPTIONS = {"""
 for key in "${!modes[@]}"; do printf "%20s => %s,\n" $key ${modes[$key]}; done
 echo """}
 RUN_DIRECTORIES = ["""
 for rdir in ${rdirs[@]}; do echo "  $rdir,"; done
 echo """]
 $sep
 """
 
 # check cache (and exit), if requested
 if ${modes['check-cache']}; then
   echo "Cross-checking /cache and /mss..."
   $TIMELINESRC/libexec/check-cache.sh ${rdirs[@]}
   exit $?
 fi
 
 # if `flatdir` mode, populate `rdirs` with the list of files, since our job loop will be over `rdirs` elements
 if ${modes['flatdir']}; then
   rdirsIn=("${rdirs[@]}") # make a copy, since it will be overwritten
   rdirs=()
   for rdir in ${rdirsIn[@]}; do
     for hipofile in $(ls $(realpath $rdir)/*.hipo); do
       rdirs+=($hipofile)
     done
   done
 fi
 
 # initial checks and preparations
 echo $dataset | grep -q "/" && printError "dataset name must not contain '/' " && echo && exit 100
 [ -z "$dataset" ] && printError "dataset name must not be empty" && echo && exit 100
 slurmJobName=clas12-timeline--$dataset
 
 # start job lists
 echo """
 Generating job scripts..."""
 slurmDir=./slurm
 mkdir -p $slurmDir/scripts
 jobkeys=()
 for key in detectors physics; do
   if ${modes['focus-all']} || ${modes['focus-'$key]}; then
     jobkeys+=($key)
   fi
 done
 declare -A joblists
 for key in ${jobkeys[@]}; do
   joblists[$key]=$slurmDir/job.$dataset.$key.list
   > ${joblists[$key]}
 done
 
 # define backup directory (used only if the output files already exist; not used `if ${modes['swifjob']}`)
 backupDir=$(pwd -P)/tmp/backup.$dataset.$(date +%s) # use unixtime for uniqueness
 
 # loop over input directories, building the job lists
 for rdir in ${rdirs[@]}; do
 
   # get the run number, either from `rdir` basename (fast), or from `RUN::config` (slow)
   [[ ! -e $rdir ]] && printError "the run file/directory '$rdir' does not exist" && continue
   runnum=$(basename $rdir | grep -m1 -o -E "[0-9]+" || echo '') # first, try from run directory (or file) basename
   if [ -z "$runnum" ] || ${modes['swifjob']}; then # otherwise, use RUN::config from a HIPO file (NOTE: assumes all HIPO files have the same run number)
     if ${modes['flatdir']}; then
       $TIMELINESRC/libexec/hipo-check.sh $rdir
       runnum=$($TIMELINESRC/libexec/run-groovy-timeline.sh $TIMELINESRC/libexec/get-run-number.groovy $rdir | tail -n1 | grep -m1 -o -E "[0-9]+" || echo '')
     else
       firstHipo=$(find $rdir -name "*.hipo" | head -n1)
       [ -z "$firstHipo" ] && printError "no HIPO files in run directory '$rdir'; cannot get run number or create job" && continue
       echo "using HIPO file $firstHipo to get run number for run directory '$rdir'"
       $TIMELINESRC/libexec/hipo-check.sh $firstHipo
       runnum=$($TIMELINESRC/libexec/run-groovy-timeline.sh $TIMELINESRC/libexec/get-run-number.groovy $firstHipo | tail -n1 | grep -m1 -o -E "[0-9]+" || echo '')
     fi
   fi
   [ -z "$runnum" -o $runnum -eq 0 ] && printError "unknown run number for '$rdir'; ignoring it!" && continue
   runnum=$((10#$runnum))
   echo "run directory/file '$rdir' has run number $runnum"
 
   # get list of input files, and append prefix for SWIF
   echo "..... getting its input files ....."
   inputListFile=$slurmDir/files.$dataset.$runnum.inputs.list
   if ${modes['flatdir']}; then
     realpath $rdir > $inputListFile
   else
     [[ "$(realpath $rdir)" =~ /mss/ ]] && swifPrefix="mss:" || swifPrefix="file:"
     realpath $rdir/*.hipo | sed "s;^;$swifPrefix;" > $inputListFile
   fi
 
   # generate job scripts
   echo "..... generating its job scripts ....."
   for key in ${jobkeys[@]}; do
 
     # preparation: make output subdirectory and backup old one, if it exists
     outputSubDir=$outputDir/timeline_$key/$runnum
     if ${modes['swifjob']}; then
       outputSubDir=$outputDir # no need for run subdirectory or backup on swif runner
     else
       if [ -d $outputSubDir ]; then
         mkdir -p $backupDir/timeline_$key
         mv -v $outputSubDir $backupDir/timeline_$key/
       fi
     fi
     mkdir -p $outputSubDir
 
     # get beam energy from RCDB
     echo "Retrieving beam energy from RCDB..."
     beam_energy=$($TIMELINESRC/libexec/get-beam-energy.sh $runnum | tail -n1)
     # override beam energy, for cases where RCDB is incorrect
     # - currently only needed for RG-F
     beam_energy_override=`python -c """
 beamlist = [
   (11620, 11657, 2.182),
   (12389, 12443, 2.182),
   (12444, 12951, 10.389),
 ]
 for r0,r1,eb in beamlist:
   if $runnum>=r0 and $runnum<=r1:
     print(eb)
     """`
     if [ -n "$beam_energy_override" ]; then
       if [ -n "$beam_energy" ]; then
         printWarning "overriding RCDB beam energy $beam_energy to be $beam_energy_override, for run $runnum"
       fi
       beam_energy=$beam_energy_override
     fi
     if [ -z "$beam_energy" ]; then
       printError "Unknown beam energy for run $runnum, since RCDB query failed and no overriding beam energy was provided"
       exit 100
     fi
     echo "Beam energy = $beam_energy"
 
     # make job scripts for each $key
     jobscript=$slurmDir/scripts/$key.$dataset.$runnum.sh
     case $key in
 
       detectors)
         cat > $jobscript << EOF
 #!/usr/bin/env bash
 set -e
 set -u
 set -o pipefail
 echo "RUN $runnum"
 
 # set env vars
 source $TIMELINESRC/libexec/environ.sh
 
 # produce histograms
 java $TIMELINE_JAVA_OPTS \\
   org.jlab.clas.timeline.histograms.run_histograms \\
     $runnum \\
     $outputSubDir \\
     $inputListFile \\
     $MAX_NUM_EVENTS \\
     $beam_energy
 
 # check output HIPO files
 $TIMELINESRC/libexec/hipo-check.sh \$(find $outputSubDir -name "*.hipo")
 EOF
         ;;
 
       physics)
         if ${modes['flatdir']}; then
           monitorReadType=skim
         else
           monitorReadType=dst
         fi
         cat > $jobscript << EOF
 #!/usr/bin/env bash
 set -e
 set -u
 set -o pipefail
 echo "RUN $runnum"
 
 # set env vars
 source $TIMELINESRC/libexec/environ.sh
 
 # produce histograms
 $TIMELINESRC/libexec/run-groovy-timeline.sh \\
   $TIMELINESRC/qa-physics/monitorRead.groovy \\
     $(realpath $rdir) \\
     $outputSubDir \\
     $monitorReadType \\
     $runnum \\
     $beam_energy
 
 # check output HIPO files
 $TIMELINESRC/libexec/hipo-check.sh \$(find $outputSubDir -name "*.hipo")
 EOF
         ;;
 
     esac
 
     # grant permission and add it `joblists`
     chmod u+x $jobscript
     echo $jobscript >> ${joblists[$key]}
 
   done # loop over `jobkeys`
 
 done # loop over `rdirs`
 
 
 # now generate slurm descriptions and/or local scripts
 echo """
 Generating batch scripts..."""
 exelist=()
 for key in ${jobkeys[@]}; do
 
   # check if we have any jobs to run
   joblist=${joblists[$key]}
   [ ! -s $joblist ] && printError "there are no $key timeline jobs to run" && continue
   slurm=$(echo $joblist | sed 's;.list$;.slurm;')
 
   # either generate single/sequential run scripts
   if ${modes['single']} || ${modes['series']} || ${modes['swifjob']}; then
     localScript=$(echo $joblist | sed 's;.list$;.local.sh;')
     echo "#!/usr/bin/env bash" > $localScript
     echo "set -e" >> $localScript
     if ${modes['single']}; then
       head -n1 $joblist >> $localScript
     else # ${modes['series']} || ${modes['swifjob']}
       cat $joblist >> $localScript
     fi
     chmod u+x $localScript
     exelist+=($localScript)
 
   # otherwise generate slurm description
   else
     cat > $slurm << EOF
 #!/bin/sh
 #SBATCH --ntasks=1
 #SBATCH --job-name=$slurmJobName--$key
 #SBATCH --output=$SLURM_LOG.out
 #SBATCH --error=$SLURM_LOG.err
 #SBATCH --partition=production
 #SBATCH --account=clas12
 
 #SBATCH --mem-per-cpu=$SLURM_MEMORY
 #SBATCH --time=$SLURM_TIME
 
 #SBATCH --array=1-$(cat $joblist | wc -l)
 #SBATCH --ntasks=1
 
 srun \$(head -n\$SLURM_ARRAY_TASK_ID $joblist | tail -n1)
 EOF
     exelist+=($slurm)
   fi
 done
 
 
 # execution
 [ ${#exelist[@]} -eq 0 ] && printError "no jobs were created at all; check errors and warnings above" && exit 100
 echo """
 $sep
 """
 if ${modes['single']} || ${modes['series']} || ${modes['swifjob']}; then
   if ${modes['single']}; then
     echo "RUNNING ONE SINGLE JOB LOCALLY:"
   elif ${modes['series']}; then
     echo "RUNNING ALL JOBS SEQUENTIALLY, LOCALLY:"
   fi
   for exe in ${exelist[@]}; do
     echo """
     $sep
     EXECUTING: $exe
     $sep"""
     $exe
   done
 elif ${modes['submit']}; then
   echo "SUBMITTING JOBS TO SLURM"
   echo $sep
   for exe in ${exelist[@]}; do sbatch $exe; done
   echo $sep
   echo "JOBS SUBMITTED!"
 else
   echo """  SLURM JOB DESCRIPTIONS GENERATED
   - Slurm job name prefix will be: $slurmJobName
   - To submit all jobs to slurm, run:
     ------------------------------------------"""
   for exe in ${exelist[@]}; do echo "    sbatch $exe"; done
   echo """    ------------------------------------------
   """
 fi

run-detectors-timelines.sh -> qtl-analysis

--- v2/bin/run-detectors-timelines.sh	2025-04-28 10:35:31.099904323 -0400
+++ v3/bin/qtl-analysis	2025-04-28 10:35:44.023832013 -0400
@@ -1,359 +1,516 @@
 #!/usr/bin/env bash
 
 set -e
 set -u
 source $(dirname $0)/../libexec/environ.sh
 
+##################################################################################
+
+# timeline webserver directory
+WEBURL="https://clas12mon.jlab.org"
+WEBDIR=/group/clas/www/clas12mon/html/hipo
+
+##################################################################################
+
 # default options
 match="^"
 inputDir=""
+publishDir=""
+publishNote=""
+metaSettings="" # FIXME: generalize `metaSettings` to run-group-specific settings (e.g., exclude irrelevant timelines for RG-L)
 dataset=""
 outputDir=""
 numThreads=8
 singleTimeline=""
+# modes, set by CLI long opts (--$key)
 declare -A modes
-for key in list skip-mya focus-timelines focus-qa debug help; do
+for key in list just-pub just-ana skip-mya debug overwrite custom-pub help; do
   modes[$key]=false
 done
 
-# input finding command
-inputCmd="$TIMELINESRC/libexec/set-input-dir.sh -s timeline_detectors"
-inputCmdOpts=""
+##################################################################################
 
 # usage
 sep="================================================================"
-usage() {
+usageTerse() {
   echo """
   $sep
-  USAGE: $0 [OPTIONS]...
+  USAGE: qtl analysis [OPTIONS]...
   $sep
-  Creates web-ready detector timelines locally
+  Analyzes histograms and produces timelines.
 
-  REQUIRED OPTIONS: specify at least one of the following:""" >&2
-  $inputCmd -h
-  echo """
-  OPTIONAL OPTIONS:
+  CHEF OPTIONS: most of these are required for chef's production
+
+    -i [INPUT_DIR]      input directory; use the output workflow
+                        directory, which has subdirectories for each run
+
+    -p [PUBLISH_DIR]    publish timeline results to URL
+                        $WEBURL/[PUBLISH_DIR]/tlsummary
+                        (i.e., $WEBDIR/[PUBLISH_DIR])
+
+    -n [NOTE]           additional note that will be shown on the clas12mon
+                        webpage; surround it in quotes
+                        - default: ''
+
+    -s [SETTINGS]       apply run-group or dataset specific settings; one of:
+$(find $TIMELINESRC/data/metadata -name "*.json" -exec basename {} .json \; | sed 's;^;                           ;')
+                        - default: no custom settings
+  """>&2
+}
+usageVerbose() {
+  echo """  DEVELOPER OPTIONS: these are generally for developers, not chefs
+
+    -d [DATASET_NAME]   unique dataset name, defined by the user
+                        - you may have used it in 'qtl histogram'
+                        - default: directory basename of [PUBLISH_DIR]
+
+    -i [INPUT_DIR]      input directory
+                        - the default is such that users only ever have
+                          to use '-d [DATASET]' to refer to a local dataset
+                        - default: ./outfiles/[DATASET_NAME]
 
-    -o [OUTPUT_DIR]     output directory
-                        default = ./outfiles/[DATASET_NAME]
+    -o [OUTPUT_DIR]     output directory for the analyzed timelines
+                        - default = ./outfiles/[DATASET_NAME]
 
-    -n [NUM_THREADS]    number of parallel threads to run
-                        default = $numThreads
+    -j [NUM_THREADS]    number of parallel threads to run
+                        - default = $numThreads
 
     -t [TIMELINE]       produce only the single detector timeline [TIMELINE]; useful for debugging
-                        use --list to dump the list of timelines
-                        default: run all
+                        - use --list to dump the list of timelines
+                        - default: run all
 
     -m [MATCH]          only produce timelines matching [MATCH]
 
     --list              dump the list of timelines and exit
 
-    --skip-mya          skip timelines which require MYA (needed if running offsite or on CI)
+    --just-ana          just do the analysis and do not attempt to publish
+                        - follow up with '--just-pub' and '-p' to publish
+
+    --just-pub          just publish analyzed timelines from [OUTPUT_DIR]
+
+    --overwrite         allow publishing to overwrite the target directory
 
-    --focus-timelines   only produce the detector timelines, do not run detector QA code
-    --focus-qa          only run the QA code (assumes you have detector timelines already)
+    --custom-pub        interpret [PUBLISH_DIR] as a fully custom directory,
+                        rather than as a subdirectory of the web server
+
+    --skip-mya          skip timelines which require MYA (needed if running offsite or on CI)
 
     --debug             enable debug mode: run a single timeline with stderr and stdout printed to screen;
                         it is best to use this with the '-t' option to debug specific timeline issues
 
     -h, --help          print this usage guide
   """ >&2
 }
 if [ $# -eq 0 ]; then
-  usage
+  usageTerse
+  echo "  For more options, run with '--help'" >&2
   exit 101
 fi
 
 # parse options
-while getopts "d:i:Uo:r:n:t:m:h-:" opt; do
+while getopts "d:i:p:n:s:o:j:t:m:h-:" opt; do
   case $opt in
-    d) inputCmdOpts+=" -d $OPTARG" ;;
-    i) inputCmdOpts+=" -i $OPTARG" ;;
-    U) inputCmdOpts+=" -U" ;;
+    d)
+      echo $OPTARG | grep -q "/" && printError "dataset name must not contain '/' " && exit 100
+      dataset=$OPTARG
+      ;;
+    i) inputDir=$OPTARG ;;
+    p) publishDir=$OPTARG ;;
+    n) publishNote=$OPTARG ;;
+    s) metaSettings=$OPTARG ;;
     o) outputDir=$OPTARG ;;
-    r) printError "option '-r' has been deprecated, since it is no longer needed" && exit 100 ;;
-    n) numThreads=$OPTARG ;;
-    m) match=$OPTARG ;;
+    j) numThreads=$OPTARG ;;
     t) singleTimeline=$OPTARG ;;
+    m) match=$OPTARG ;;
     h) modes['help']=true ;;
     -)
       for key in "${!modes[@]}"; do
         [ "$key" == "$OPTARG" ] && modes[$OPTARG]=true && break
       done
       [ -z "${modes[$OPTARG]-}" ] && printError "unknown option --$OPTARG" && exit 100
       ;;
     *) exit 100;;
   esac
 done
 if ${modes['help']}; then
-  usage
+  usageTerse
+  usageVerbose
   exit 101
 fi
 
-# get main executable for detector timelines
+##################################################################################
+
+# main executable for detector timelines
 run_analysis_script="org.jlab.clas.timeline.analysis.run_analysis"
 
 # build list of timelines
 if ${modes['skip-mya']}; then
   timelineList=$(java $TIMELINE_JAVA_OPTS $run_analysis_script --timelines | grep -vE '^epics_' | sort | grep $match)
 else
   timelineList=$(java $TIMELINE_JAVA_OPTS $run_analysis_script --timelines | sort | grep $match)
 fi
 
 # list detector timelines, if requested
 if ${modes['list']}; then
   echo $sep
   echo "LIST OF TIMELINES"
   echo $sep
   echo $timelineList | sed 's; ;\n;g'
   exit $?
 fi
 
-# set input/output directories and dataset name
-dataset=$($inputCmd $inputCmdOpts -D)
-inputDir=$(realpath $($inputCmd $inputCmdOpts -I))
+##################################################################################
+
+# set flow control booleans
+## handle `--just-ana` and `--just-pub`
+enableAna=false
+enablePub=false
+if ${modes['just-ana']} && ! ${modes['just-pub']}; then # --just-ana
+  enableAna=true
+  enablePub=false
+elif ! ${modes['just-ana']} && ${modes['just-pub']}; then # --just-pub
+  enableAna=false
+  enablePub=true
+elif ${modes['just-ana']} && ${modes['just-pub']}; then # --just-ana AND --just-pub
+  printWarning "both '--just-ana' and '--just-pub' means that we will do both analysis and publishing, i.e., the default behavior"
+  enableAna=true
+  enablePub=true
+else # default behavior (neither --just-ana NOR --just-pub)
+  enableAna=true
+  enablePub=true
+fi
+## disable publishing when `--debug`
+if ${modes['debug']}; then
+  printWarning "DEBUG mode used; timelines will NOT be published"
+  enableAna=true
+  enablePub=false
+fi
+
+##################################################################################
+
+# check publishing options
+publishUrl=""
+if $enablePub; then
+  # check publishDir arg
+  if [ -z "$publishDir" ]; then
+    printError "option '-p' is required, to specify a publishing directory"
+    printError "(otherwise, use '--just-ana' if you really do not want to publish)"
+    exit 100
+  fi
+  # set full path to publish dir (unless --custom-pub)
+  if ! ${modes['custom-pub']}; then
+    publishUrl=$WEBURL/$publishDir/tlsummary
+    [ "$publishDir" = "." ] && printError "publishing directory argument cannot be '.'" && exit 100
+    [[ "$publishDir" =~ '..' ]] && printError "publishing directory cannot contain '..'" && exit 100
+    publishDir=$WEBDIR/$publishDir
+  else
+    publishUrl="option '--custom-pub' was used, therefore timeline URL is UNKNOWN"
+  fi
+  echo "publishing directory: $publishDir"
+  # check if the publishing directory already exists
+  if [ -d $publishDir ]; then
+    if ${modes['overwrite']}; then
+      printWarning "Publishing directory already exists! Timelines will be overwritten!"
+      printWarning "Press 'Ctrl-C' NOW if this is not what you want! Otherwise, just wait..."
+      sleep 5
+      rm -rv $publishDir
+    else
+      printError "Publishing directory already exists! Either choose another directory, or re-run with '--overwrite' option to overwrite (be careful...)"
+      exit 100
+    fi
+  fi
+  # make the publishing directory (which also verifies write permission)
+  mkdir -pv $publishDir
+  # path to filter file
+  if [ -n "$metaSettings" ]; then
+    metaSettings=$TIMELINESRC/data/metadata/$metaSettings.json
+    if [ -f "$metaSettings" ]; then
+      echo "using metadata file '$metaSettings'"
+    else
+      printError "metadata file '$metaSettings' does not exist (bad argument for option '-s')"
+      exit 100
+    fi
+  fi
+  # add info to the note
+  version=$($TIMELINESRC/bin/qtl --version)
+  timestamp=$(date '+%a %D')
+  [ -n "$publishNote" ] && publishNote="; $publishNote"
+  publishNote="${timestamp}${publishNote}; timeline-code v$version"
+fi
+
+# set the dataset name, if not already set
+if [ -z "$dataset" ]; then
+  if [ -n "$publishDir" ]; then
+    dataset=$(basename $publishDir)
+    echo "setting dataset name from the publishing directory to '$dataset'"
+  else
+    printError "neither dataset name (-d) nor publishing directory (-p) are set; need at least one..."
+    exit 100
+  fi
+fi
+
+# set input and output directories
+[ -z "$inputDir" ] && inputDir=$(pwd -P)/outfiles/$dataset/timeline_detectors
+[ ! -d $inputDir ] && printError "input directory $inputDir does not exist" && exit 100
+inputDir=$(realpath $inputDir)
 [ -z "$outputDir" ] && outputDir=$(realpath $(pwd -P)/outfiles/$dataset) || outputDir=$(realpath $outputDir)
 
 # set subdirectories
 finalDirPreQA=$outputDir/timeline_web_preQA
 finalDir=$outputDir/timeline_web
 logDir=$outputDir/log
 
-# check focus options
-modes['focus-all']=true
-for key in focus-timelines focus-qa; do
-  if ${modes[$key]}; then modes['focus-all']=false; fi
-done
-
 # print settings
-echo """
+if $enableAna; then
+  echo """
 Settings:
 $sep
 INPUT_DIR       = $inputDir
 DATASET_NAME    = $dataset
 OUTPUT_DIR      = $outputDir
 FINAL_DIR_PREQA = $finalDirPreQA
 FINAL_DIR       = $finalDir
 LOG_DIR         = $logDir
+PUBLISH_DIR     = $publishDir
 NUM_THREADS     = $numThreads
 OPTIONS = {"""
-for key in "${!modes[@]}"; do printf "%20s => %s,\n" $key ${modes[$key]}; done
-echo "}"
+  for key in "${!modes[@]}"; do printf "%20s => %s,\n" $key ${modes[$key]}; done
+  echo "}"
+fi
+
 
 # output detector subdirectories
 detDirs=(
   alert
   band
   bmtbst
   central
   cnd
   ctof
   cvt
   dc
   ec
   epics
   forward
   ft
   ftof
   helicity
   htcc
   ltcc
   # m2_ctof_ftof
   rf
   rich
   trigger
 )
 
 # cleanup output directories
-if ${modes['focus-all']} || ${modes['focus-timelines']}; then
+if $enableAna; then
   if [ -d $finalDirPreQA ]; then
-    rm -rv $finalDirPreQA
+    printWarning "removing output directory $finalDirPreQA"
+    rm -r $finalDirPreQA
+  fi
+  if [ -d $logDir ]; then
+    printWarning "removing log directory $logDir"
+    rm -r $logDir
   fi
-fi
-if [ -d $logDir ]; then
-  for fail in $(find $logDir -name "*.fail"); do
-    rm $fail
-  done
 fi
 
 # make output directories
 mkdir -p $logDir $finalDirPreQA $finalDir
 
-function wait_for_jobs() {
-    stat=10
-    while [ "${#job_ids[@]}" -gt $1 ]; do
-        for i in "${!job_ids[@]}"; do
-            if [ "$1" -eq 0 ]; then
-                if [ "${#job_ids[@]}" -lt $stat ]; then
-                    echo ">>> $(date) >>> waiting on ${#job_ids[@]} jobs:"
-                    let stat=${#job_ids[@]}
-                    #let stat=stat-1
-                    printf '>>>     %s\n' "${job_names[@]}"
-                fi
-            fi
-            set +e
-            ps ${job_ids[$i]} >& /dev/null
-            if [ "$?" -ne 0 ]; then
-                echo ">>> ${job_names[$i]} finished."
-                unset job_ids[$i]
-                unset job_names[$i]
-            fi
-            set -e
-        done
-        sleep 1
-    done
-}
+
+##################################################################################
+##################################################################################
+##################################################################################
+
 
 ######################################
 # produce detector timelines
 ######################################
-if ${modes['focus-all']} || ${modes['focus-timelines']}; then
+if $enableAna; then
+
+  function wait_for_jobs() {
+      stat=10
+      while [ "${#job_ids[@]}" -gt $1 ]; do
+          for i in "${!job_ids[@]}"; do
+              if [ "$1" -eq 0 ]; then
+                  if [ "${#job_ids[@]}" -lt $stat ]; then
+                      echo ">>> $(date) >>> waiting on ${#job_ids[@]} jobs:"
+                      let stat=${#job_ids[@]}
+                      #let stat=stat-1
+                      printf '>>>     %s\n' "${job_names[@]}"
+                  fi
+              fi
+              set +e
+              ps ${job_ids[$i]} >& /dev/null
+              if [ "$?" -ne 0 ]; then
+                  echo ">>> ${job_names[$i]} finished."
+                  unset job_ids[$i]
+                  unset job_names[$i]
+              fi
+              set -e
+          done
+          sleep 1
+      done
+  }
 
   # change working directory to output directory
   pushd $finalDirPreQA
 
   # make detector subdirectories
   for detDir in ${detDirs[@]}; do
     mkdir -p $detDir
   done
 
   # produce timelines, multithreaded
   job_ids=()
   job_names=()
   for timelineObj in $timelineList; do
     logFile=$logDir/$timelineObj
     [ -n "$singleTimeline" -a "$timelineObj" != "$singleTimeline" ] && continue
     echo ">>> producing timeline '$timelineObj' ..."
     if ${modes['debug']}; then
       java $TIMELINE_JAVA_OPTS $run_analysis_script $timelineObj $inputDir
       echo "PREMATURE EXIT, since --debug option was used"
       exit
     else
       java $TIMELINE_JAVA_OPTS $run_analysis_script  $timelineObj $inputDir > $logFile.out 2> $logFile.err || touch $logFile.fail &
       job_ids+=($!)
       job_names+=($timelineObj)
     fi
     wait_for_jobs $numThreads
   done
 
   wait_for_jobs 0
 
   # organize output timelines
   echo ">>> organizing output timelines..."
   timelineFiles=$(find -name "*.hipo")
   [ -z "$timelineFiles" ] && printError "no timelines were produced; check error logs in $logDir/" && exit 100
   for timelineFile in $timelineFiles; do
     det=$(basename $timelineFile .hipo | sed 's;_.*;;g')
     case $det in
       bmt)    mv $timelineFile bmtbst/  ;;
       bst)    mv $timelineFile bmtbst/  ;;
       cen)    mv $timelineFile central/ ;;
       ftc)    mv $timelineFile ft/      ;;
       fth)    mv $timelineFile ft/      ;;
       rat)    mv $timelineFile trigger/ ;;
       rftime) mv $timelineFile rf/      ;;
       # ctof|ftof)
       #   [[ "$timelineFile" =~ _m2_ ]] && mv $timelineFile m2_ctof_ftof/ || mv $timelineFile $det/
       #   ;;
       *)
         if [ -d $det ]; then
           mv $timelineFile $det/
         else
           printError "not sure where to put timeline '$timelineFile' for detector '$det'; please update $0 to fix this" && exit 100
         fi
         ;;
     esac
   done
 
   # check timelines; remove and complain about any bad ones
   echo ">>> running hipo-check on timeline HIPO files..."
   outputFiles=$(find . -name "*.hipo")
   if [ -n "$outputFiles" ]; then
     logFile=$logDir/hipo-check
     $TIMELINESRC/libexec/hipo-check.sh --rm-bad $outputFiles > $logFile.out 2> $logFile.err || touch $logFile.fail
   fi
 
   # remove any empty directories
   echo ">>> removing any empty directories..."
   for detDir in ${detDirs[@]}; do
     [ -z "$(find $detDir -name '*.hipo')" ] && rm -rv $detDir
   done
 
   echo ">>> done producing timelines..."
   popd
 fi
 
 ######################################
 # run QA
 ######################################
 
-# first, copy the timelines to the final timeline directory; we do this regardless of whether QA is run
-# so that (1) only `$finalDir` needs deployment and (2) we can re-run the QA with 'focus-qa' mode
-echo ">>> copy timelines to final directory..."
-cp -rL $finalDirPreQA/* $finalDir/
-
-if ${modes['focus-all']} || ${modes['focus-qa']}; then
+if $enableAna; then
+  # first, copy the timelines to the final timeline directory
+  echo ">>> copy timelines to final directory..."
+  cp -rL $finalDirPreQA/* $finalDir/
+  # then add the QA lines
   echo ">>> add QA lines..."
   logFile=$logDir/qa
   $TIMELINESRC/libexec/run-groovy-timeline.sh $TIMELINESRC/qa-detectors/util/applyBounds.groovy $finalDirPreQA $finalDir > $logFile.out 2> $logFile.err || touch $logFile.fail
   outputFiles=$(find $finalDir -name "*_QA.hipo")
-  [ -n "$outputFiles" ] && $TIMELINESRC/libexec/hipo-check.sh $outputFiles
+  logFile=$logDir/qa_hipo-check
+  if [ -n "$outputFiles" ]; then
+    $TIMELINESRC/libexec/hipo-check.sh $outputFiles > $logFile.out 2> $logFile.err || touch $logFile.fail
+  fi
 fi
 
 
+
 ######################################
 # error checking
 ######################################
 
-
-# print log file info
-echo """
-$sep
-OUTPUT AND ERROR LOGS:
-$logDir/*.out
-$logDir/*.err
-"""
-
-# exit nonzero if any jobs exitted nonzero
-failedJobs=($(find $logDir -name "*.fail" | xargs -I{} basename {} .fail))
 somethingFailed=false
-if [ ${#failedJobs[@]} -gt 0 ]; then
-  for failedJob in ${failedJobs[@]}; do
-    echo $sep
-    printError "job '$failedJob' returned non-zero exit code; error log dump:"
-    cat $logDir/$failedJob.err
-    if [ "$failedJob" = "hipo-check" ]; then
-      printWarning "These HIPO files are TIMELINE files; if this '$failedJob' job is the ONLY failed job, you may proceed with timeline deployment, but these failed timelines will not be deployed."
-    fi
-  done
-  if [ -z "$singleTimeline" -a ${modes['focus-qa']} = false ]; then
-    echo $sep
-    echo "To re-run only the failed timelines, for debugging, try one of the following commands:"
+if $enableAna; then
+
+  # print log file info
+  echo """
+  $sep
+  OUTPUT AND ERROR LOGS:
+  $logDir/*.out
+  $logDir/*.err
+  """
+
+  # exit nonzero if any jobs exitted nonzero
+  failedJobs=($(find $logDir -name "*.fail" | xargs -I{} basename {} .fail))
+  if [ ${#failedJobs[@]} -gt 0 ]; then
     for failedJob in ${failedJobs[@]}; do
-      if [ "$failedJob" = "qa" ]; then
-        echo "  $0 $@ --focus-qa"
-      elif [ "$failedJob" = "hipo-check" ]; then
-        echo "  $0 $@ --focus-timelines -t [BAD_TIMELINE]"
-        echo "  where [BAD_TIMELINE] is any timeline that failed 'hipo-check'"
-      else
-        echo "  $0 $@ --focus-timelines -t $failedJob"
+      echo $sep
+      printError "job '$failedJob' returned non-zero exit code; error log dump:"
+      cat $logDir/$failedJob.err
+      if [ "$failedJob" = "hipo-check" ]; then
+        printWarning "These HIPO files are TIMELINE files; they will NOT be published"
       fi
     done
+    somethingFailed=true
+  else
+    echo "All jobs exitted normally"
   fi
-  somethingFailed=true
-else
-  echo "All jobs exitted normally"
+
+  # grep for suspicious things in error logs
+  errPattern="error:|exception:|warning"
+  echo "Now scanning for any quieter errors, by running \`grep -iE '$errPattern'\` on *.err files:"
+  echo $sep
+  grep -iE --color "$errPattern" $logDir/*.err && printWarning "errors/warnings found in log files!" || echo "Good news: grep found no errors, but you still may want to take a look yourself..."
+  echo $sep
+
 fi
 
-# grep for suspicious things in error logs
-errPattern="error:|exception:|warning"
-echo """Now scanning for any quieter errors, by running \`grep -iE '$errPattern'\` on *.err files:
-$sep"""
-grep -iE --color "$errPattern" $logDir/*.err || echo "Good news: grep found no errors, but you still may want to take a look yourself..."
-echo $sep
+
+######################################
+# publishing
+######################################
+
+if $enablePub; then
+  echo "Publishing..."
+  cp -r $finalDir/* $publishDir/
+  [ -n "$metaSettings" ] && cp -v $metaSettings $publishDir/metadata.json
+  [ -n "$publishNote" ] && echo "$publishNote" > $publishDir/README
+  $TIMELINESRC/libexec/run-groovy-timeline.sh $TIMELINESRC/libexec/index-webpage.groovy $publishDir
+  echo """
+Done! Timeline files published to: $publishDir
+___________________________________________________________
+TIMELINE URL:
+  $publishUrl
+___________________________________________________________
+  """
+fi
 
 # exit nonzero if something failed
 if $somethingFailed; then
   printWarning "At least one job had issues; look above or in the log files to see what's wrong."
   exit 100
 fi

run-physics-timelines.sh -> qtl-physics

--- v2/bin/run-physics-timelines.sh	2025-04-28 10:35:31.100904313 -0400
+++ v3/bin/qtl-physics	2025-04-28 10:35:44.023832013 -0400
@@ -1,141 +1,179 @@
 #!/usr/bin/env bash
 
 set -e
 set -u
 source $(dirname $0)/../libexec/environ.sh
 
 # default options
 inputDir=""
 dataset=""
 outputDir=""
-
-# input finding command
-inputCmd="$TIMELINESRC/libexec/set-input-dir.sh -s timeline_physics"
-inputCmdOpts=""
+publishDir=""
+# modes, set by CLI long opts (--$key)
+declare -A modes
+for key in custom-pub help; do
+  modes[$key]=false
+done
 
 # usage
 sep="================================================================"
 usage() {
   echo """
   $sep
   USAGE: $0 [OPTIONS]...
   $sep
   Creates web-ready physics timelines locally
 
-  REQUIRED OPTIONS: specify at least one of the following:""" >&2
-  $inputCmd -h
-  echo """
+  REQUIRED OPTIONS: specify at least one of the following:
+
+    -d [DATASET_NAME]   unique dataset name, defined by the user
+
+    -i [INPUT_DIR]      input directory
+                        default = ./outfiles/[DATASET_NAME]
+
+    -p [PUBLISH_DIR]    publish timeline results (see 'qtl analysis');
+                        use --custom-pub for fully custom dir
+
   OPTIONAL OPTIONS:
 
     -o [OUTPUT_DIR]     output directory
                         default = ./outfiles/[DATASET_NAME]
 
     -h, --help          print this usage guide
   """ >&2
 }
 if [ $# -eq 0 ]; then
   usage
   exit 101
 fi
 
 # parse options
-helpMode=false
-while getopts "d:i:Uo:h-:" opt; do
+while getopts "d:i:p:o:h-:" opt; do
   case $opt in
-    d) inputCmdOpts+=" -d $OPTARG" ;;
-    i) inputCmdOpts+=" -i $OPTARG" ;;
-    U) inputCmdOpts+=" -U" ;;
+    d)
+      echo $OPTARG | grep -q "/" && printError "dataset name must not contain '/' " && exit 100
+      dataset=$OPTARG
+      ;;
+    i) inputDir=$OPTARG ;;
     o) outputDir=$OPTARG ;;
-    h) helpMode=true ;;
+    p) publishDir=$OPTARG ;;
+    h) modes['help']=true ;;
     -)
-      [ "$OPTARG" != "help" ] && printError "unknown option --$OPTARG"
-      helpMode=true
+      for key in "${!modes[@]}"; do
+        [ "$key" == "$OPTARG" ] && modes[$OPTARG]=true && break
+      done
+      [ -z "${modes[$OPTARG]-}" ] && printError "unknown option --$OPTARG" && exit 100
       ;;
   esac
 done
-if $helpMode; then
+if ${modes['help']}; then
   usage
   exit 101
 fi
 
-# set input/output directories and dataset name
-dataset=$($inputCmd $inputCmdOpts -D)
-inputDir=$(realpath $($inputCmd $inputCmdOpts -I))
+# set input and output directories
+[ -z "$dataset" ] && printError "data set not specified, use option '-d'" && exit 100
+[ -z "$inputDir" ] && inputDir=$(pwd -P)/outfiles/$dataset/timeline_physics
+[ ! -d $inputDir ] && printError "input directory $inputDir does not exist" && exit 100
+inputDir=$(realpath $inputDir)
 [ -z "$outputDir" ] && outputDir=$(realpath $(pwd -P)/outfiles/$dataset) || outputDir=$(realpath $outputDir)
+[ -z "$publishDir" ] && printError "need publishing directory, option '-p'" && exit 100
 
 # set subdirectories
 qaDir=$outputDir/timeline_physics_qa
 finalDir=$outputDir/timeline_web
-logDir=$outputDir/log
+logDir=$outputDir/log_physics
 
 # print settings
 echo """
 Settings:
 $sep
 INPUT_DIR       = $inputDir
 DATASET_NAME    = $dataset
 OUTPUT_DIR      = $outputDir
 FINAL_DIR       = $finalDir
 LOG_DIR         = $logDir
 """
 
 pushd $TIMELINESRC/qa-physics
 
 # setup error-filtered execution function
 mkdir -p $logDir
 logFile=$logDir/physics.err
 logTmp=$logFile.tmp
 > $logFile
 function exe {
   echo $sep
   echo "EXECUTE: $*"
   echo $sep
   $* 2> >(tee $logTmp >&2)
   mv $logTmp{,.bak}
   cat $logTmp.bak |\
     { grep -v '^Picked up _JAVA_OPTIONS:' || test $? = 1; } |\
     { grep -v 'VariableMetricBuilder: no improvement' || test $? = 1; } \
     > $logTmp
   rm $logTmp.bak
   if [ -s $logTmp ]; then
     echo "stderr from command:  $*" >> $logFile
     cat $logTmp >> $logFile
     echo $sep >> $logFile
   fi
 }
 
 # organize the data into datasets
 exe ./datasetOrganize.sh $dataset $inputDir $qaDir
 
 # produce chargeTree.json
 exe $TIMELINESRC/libexec/run-groovy-timeline.sh buildChargeTree.groovy $qaDir
 
 # loop over datasets
 # trigger electrons monitor
 exe $TIMELINESRC/libexec/run-groovy-timeline.sh qaPlot.groovy $qaDir
 exe $TIMELINESRC/libexec/run-groovy-timeline.sh qaCut.groovy $qaDir $dataset
 # FT electrons
 exe $TIMELINESRC/libexec/run-groovy-timeline.sh qaPlot.groovy $qaDir FT
 exe $TIMELINESRC/libexec/run-groovy-timeline.sh qaCut.groovy $qaDir $dataset FT
 # meld FT and FD JSON files
 exe $TIMELINESRC/libexec/run-groovy-timeline.sh mergeFTandFD.groovy $qaDir
 # general monitor
 exe $TIMELINESRC/libexec/run-groovy-timeline.sh monitorPlot.groovy $qaDir
 # move timelines to output area
 exe ./stageTimelines.sh $qaDir $finalDir
 # trash empty files
 exe $TIMELINESRC/libexec/run-groovy-timeline.sh $TIMELINESRC/qa-physics/removeEmptyFiles.groovy $outputDir/trash $finalDir
 
 popd
 
 # print errors
 echo """
 
 """
+hasErrorPrintouts=false
 if [ -s $logFile ]; then
   printError "some scripts had errors or warnings; dumping error output:"
   echo $sep >&2
   cat $logFile >&2
+  hasErrorPrintouts=true
 else
   echo "No errors or warnings!"
 fi
+
+# FIXME: removed the independent 'deploy-timelines.sh' script, so just call
+# qtl-analysis to do the deployment; eventually we'll absorb 'qtl-physics' into 'qtl-analysis',
+# so this kluge is good enough for now...
+echo "Now publishing..."
+customPubArg=''
+if ${modes['custom-pub']}; then
+  customPubArg=--custom-pub
+fi
+$TIMELINESRC/bin/qtl analysis \
+  -d $dataset \
+  -i $inputDir \
+  -o $outputDir \
+  --just-pub \
+  -p $publishDir $customPubArg
+
+if $hasErrorPrintouts; then
+  printWarning "At least one job complained; look above or in the log files to see what's wrong."
+  printWarning "Exiting 0, since these may just be benign warnings" ### FIXME!
+fi

Details

New top-level script qtl runs everything; it uses subcommands which call other scripts (named qtl-${subcommand}).

qtl            # show usage guide
qtl --version  # print the timeline code version

The main user-level control scripts have therefore been renamed (to qtl-*) and redesigned a bit:

Old Command	New Command
run-monitoring.sh	qtl histogram; we preserve symlink run-monitoring.sh to not break clas12-workflow; usage remains the same
run-detectors-timelines.sh	qtl analysis; ⚠️ usage has CHANGED, see below
run-physics-timelines.sh	qtl physics; ⚠️ usage has CHANGED, see below
deploy-timelines.sh	⚠️ REMOVED; qtl analysis and qtl physics now handle deployment, see below
error-print.sh	qtl error

• qtl analysis and qtl physics now handle timeline deployment ("publishing") via a new option -p, which specifies the publishing directory
  • this is different than the deploy-timelines.sh option -t, which was a "target" directory and required an additional dataset name with option -d, so the full publishing directory would be $target/$dataset
  • this is supposed to be easier for users so publishing happens right away
  • there are developer options provided to disable publishing, for example

In general, chefs should just have to:

1. run the workflow with "qtl" model
2. run qtl analysis with (at least) options -i and -p

On the other hand, developers would run:

1. qtl histogram -d some_dataset_name ...
2. qtl analysis -p $LOGNAME/some_dataset_name (to publish to their personal test web server directory)