DF/data4es: allow skipping stages

[mgolosova]

Main changes:

1. --ignore <stageID>[,<stageID>] option to skip specific stages.
2. --list option to show list of stage IDs (to be used with --ignore).
3. --help option to show usage info.
4. Minor improvements (e.g. show usage info in case of unknown option).
5. Lots of refactoring:
   • wrap pieces of code into functions;
   • reorder some actions;
   • get rid of hard-coded pid file name.
6. Readability improvements (comments, etc.).

---

The goal is to allow data4es to utilize ES update functionality when load data.
It will allow to "disable" some ETL process nodes for faster processing and not to lose information written to ES earlier. Of course, it will make high load on the ES server, so must be used with caution; still in some situations it looks like a better solution (not to load external sources; not to request ES for known information if external source is not available or doesn`t have requested information).

---

ToDo:

• make Stage 19 to recognize and properly handle "update" records (see DF/019: introduce 'update' action (ES updates, pt.1.1) #254);
• add option --update to Stage 19 (changing default action from index to update) (see DF/019: introduce 'update' action (ES updates, pt.1.1) #254);
• (make stages that might stumble over external sources mark output messages "for update":)
  • (025_chicagoES (see DF/025: skip init (ES updates, pt.2.2a) #261);)
  • (091_datasetsRucio (see DF/091: skip mode (ES updates, pt.2.2b) #262);)
  • (095_datasetInfoAMI;)
• allow to disable specific stages in data4es-start:
  • add --skip option to stages (see pyDKB: processor --skip option (ES updates, pt.2.1). #260):
    • 025_chicagoES (see DF/025: skip init (ES updates, pt.2.2a) #261);
    • 091_datasetsRucio (see DF/091: skip mode (ES updates, pt.2.2b) #262);
    • 095_datasetInfoAMI (see DF/095: skip init (ES updates, pt.2.2c) #263);
  • add --ignore STAGE... option to data4es-start (currently in this PR);
• track DF/019: EOM handling improvement #251 and pull its changes here as soon as it is approved/merged to master.

[mgolosova]

Instead of rebasing this branch on the current version of master, rewrote its history to remove "dirt" commits (appeared due to later changes in branches, that were previously merged here). Not a super-beautiful decision -- the price to pay for rebase-updates made in other branches (and for the desire to keep the merge history).

[mgolosova]

Resolve [WIP] status for what`s left to do does not really belong here -- it is up to every individual stage.

[Evildoor]

9f5b26b: this commit's description talks about changing "OC" to "09", but also transforms (unrelated to OC/09) line 208 into three lines (due to length exceeding 80 symbols, I presume). Shouldn't this change be a separate commit, or amendment to one of the previous ones?

0e4e8c9 and fb6c387: options --help and --list are implied, both by general logic and description, to merely output some info and exit. However, some work is actually done and, for example, checking of 095's credentials may fail and lead to script termination without info being displayed.

[mgolosova]

@Evildoor,

9f5b26b: this commit's description talks about changing "OC" to "09", but also transforms (unrelated to OC/09) line 208 into three lines (due to length exceeding 80 symbols, I presume).

Thank you, fixed now.

0e4e8c9 and fb6c387: options --help and --list are implied, both by general logic and description, to merely output some info and exit. However, some work is actually done and, for example, checking of 095's credentials may fail and lead to script termination without info being displayed.

Good point, thank you; it also appeared that they failed to be properly handled in case of already running process. Fixed now.

[Evildoor]

0e4e8c9 and fb6c387: options --help and --list are implied, both by general logic and description, to merely output some info and exit. However, some work is actually done and, for example, checking of 095's credentials may fail and lead to script termination without info being displayed.

Good point, thank you; it also appeared that they failed to be properly handled in case of already running process. Fixed now.

Branches are still created (Line 195) when these options are used, which may lead to similar problems:

[vaulov@aiatlas171 run]$ ./data4es-start --help
mkfifo: cannot create fifo Б─≤/afs/cern.ch/user/v/vaulov/Work/dkb/Utils/Dataflow/run/.data4es/.pipe/b_91Б─≥: Operation not permitted
(ERROR) 11-11-2019 15:05:05 (data4es) branch(): Failed to create named pipe for branch b_91

[mgolosova]

@Evildoor,

0e4e8c9 and fb6c387: options --help and --list are implied, both by general logic and description, to merely output some info and exit. However, some work is actually done and, for example, checking of 095's credentials may fail and lead to script termination without info being displayed.

Good point, thank you; it also appeared that they failed to be properly handled in case of already running process. Fixed now.

Branches are still created (Line 195) when these options are used, which may lead to similar problems:

[vaulov@aiatlas171 run]$ ./data4es-start --help
mkfifo: cannot create fifo Б─≤/afs/cern.ch/user/v/vaulov/Work/dkb/Utils/Dataflow/run/.data4es/.pipe/b_91Б─≥: Operation not permitted
(ERROR) 11-11-2019 15:05:05 (data4es) branch(): Failed to create named pipe for branch b_91

Should work properly now, please take a look.

[Evildoor]

@Evildoor,

0e4e8c9 and fb6c387: options --help and --list are implied, both by general logic and description, to merely output some info and exit. However, some work is actually done and, for example, checking of 095's credentials may fail and lead to script termination without info being displayed.

Good point, thank you; it also appeared that they failed to be properly handled in case of already running process. Fixed now.

Branches are still created (Line 195) when these options are used, which may lead to similar problems:

[vaulov@aiatlas171 run]$ ./data4es-start --help
mkfifo: cannot create fifo Б─≤/afs/cern.ch/user/v/vaulov/Work/dkb/Utils/Dataflow/run/.data4es/.pipe/b_91Б─≥: Operation not permitted
(ERROR) 11-11-2019 15:05:05 (data4es) branch(): Failed to create named pipe for branch b_91

Should work properly now, please take a look.

Yes, works properly now, thank you.

[Evildoor]

Checked again: while --help works correctly, --list still does some work and produces errors:

[vaulov@aiatlas171 run]$ ./data4es-start -l
./data4es-start: line 116: source: filename argument required
source: usage: source filename [arguments]
(ERROR) 23-01-2020 13:30:22 (data4es) Stage 095: wrong authentication parameters (failed to read certificate files:  and )
09
16
17
19
25
69
91_in
91_out
93
95

Obvious difference between --help and --list is definition of stages used in the latter to get the stage numbers from variable names, which seems to be the source of the problem (aforementioned line 116).

Similar problem arises if stage 095 is added to ignore list. Also, the error message part about "certificate files: and " can be puzzling - maybe add a specific error message for the case when files are not defined?

[Evildoor]

Comment to the PR text itself: it seems to me that four boxes starting with "make stages that might ..." should be ticked (and the last one should also link to #263) because #260, #261, #262 and #263 responsible for them were merged successfully. I'm writing this instead of fixing the text myself or ignoring it, in case that I'm wrong and something is actually not finished there.

[mgolosova]

@Evildoor,

Checked again: while --help works correctly, --list still does some work and produces errors:

[vaulov@aiatlas171 run]$ ./data4es-start -l
./data4es-start: line 116: source: filename argument required
source: usage: source filename [arguments]
(ERROR) 23-01-2020 13:30:22 (data4es) Stage 095: wrong authentication parameters (failed to > read certificate files:  and )
<...>

Similar problem arises if stage 095 is added to ignore list.
Also, the error message part about "certificate files: and " can be puzzling - maybe add a specific error message for the case when files are not defined?

I suggest to fix it in a probably unexpected way: remove handling of the files required for Stage 095 from data4es-start (check #314). This way no problem will be detected unless the stage is started -- and as a stage started, it knows what to do with these parameters in both modes ("skip" and normal processing).

[Evildoor]

Checked again: while --help works correctly, --list still does some work and produces errors:

[vaulov@aiatlas171 run]$ ./data4es-start -l
./data4es-start: line 116: source: filename argument required
source: usage: source filename [arguments]
(ERROR) 23-01-2020 13:30:22 (data4es) Stage 095: wrong authentication parameters (failed to > read certificate files:  and )
<...>

Similar problem arises if stage 095 is added to ignore list.
Also, the error message part about "certificate files: and " can be puzzling - maybe add a specific error message for the case when files are not defined?

I suggest to fix it in a probably unexpected way: remove handling of the files required for Stage 095 from data4es-start (check #314). This way no problem will be detected unless the stage is started -- and as a stage started, it knows what to do with these parameters in both modes ("skip" and normal processing).

Looks good to me.

[mgolosova]

@Evildoor,

I suggest to fix it <...> (check #314).

Looks good to me.

Alright, #314 is merged now and this PR is updated.
Looks like all the issues are addressed now, please (re-)review.

Thank you!