kmer-diversity spawns parallel jobs externally from the parallel framework

[gregcaporaso]

I was running kmer-diversity on a system with 32 processors with the intention of it running as a serial job:

qiime boots kmer-diversity     --i-table hec_table_ms5.qza     --i-sequences representative_sequences.qza     --m-metadata-file metadata.tsv     --p-sampling-depth 90816     --p-n 100     --p-no-replacement     --p-alpha-average-method median     --p-beta-average-method medoid     --output-dir kmer-90k-ms5

At multiple times during the job, it spiked CPU usage across the whole system (see screenshot). We checked, and there are no new parsl logs in runinfo/, so this seems to be independent of Parsl / the formal parallelization framework. Everything completed successfully (and fast!) but we need to track down where the parallelization is coming in so we can put that under user control.

[ebolyen]

This is most likely an issue with q2cli, but it's definitely weird.

[ebolyen]

Could it be scikit-bio? We might try fussing with OMP_NUM_THREADS and see if that helps. If so, the issue is probably in q2-diversity somewhere.

[Oddant1]

Diversity was my first thought because diversity-lib has a ton of methods that take a threads or jobs param, but that would suggest a bug with that thread param functionality which I doubt exists.

The OMP_NUM_THREADS seems like a decent idea to me

[gregcaporaso]

Thanks for the input. Are you thinking set OMP_NUM_THREADS=1 and run it again, and just see if we see a difference?

[Oddant1]

Yeah, that might be a good next thing to try