List of tools that fail containerized tool tests

[mvdbeek]

Hello IUC,

Tl;DR:

Help us fix tool tests!

we would like to make sure that our tools are well tested and working in the widest range of deployment scenarios possible. And as you may have noticed, tool dependency installation from Conda may break (or be fixed!) as time passes due to dependency-of-a-dependency upgrades, newer/older Conda versions etc.

To make sure that our tool wrappers will work on a wide range of deployments it is important that we include all dependencies that a tool has (i.e include coreutils if a wrapper uses paste, include sed if a wrapper uses sed, include python if a wrapper calls a python script). One way to make sure this is the case is to run tool tests in a minimal container that does not contain any software except the one listed in our tool wrappers <requirements/> tag set.

We also currently build containers for all IUC tools as well as other tool repositories listed in https://github.com/galaxyproject/planemo-monitor, but so far we are not testing that tool runs actually succeed in these containers. For dependencies (and dependency combinations) that are not available as a downloadable container Galaxy can build containers when needed. This functionality is also available via planemo test --biocontainers.

We have set up a github workflow that tests all IUC tools using planemo test and the --biocontainers flag. The test run can be seen here: https://github.com/mvdbeek/tools-iuc/commit/529a26943046ca80b18bccae31021db5c317eb7e/checks?check_suite_id=331921644
And a html report of only the failing tests is here: https://mvdbeek.github.io/iuc-tool-test-results/
Note also that the tool tests run against python 3.7, so this could be another class of errors we may need to fix (should concern only the cheetah templating).

The list of tools with failing tests is the following:

• multigps @gregvonkuster - Fix MultiGPS tool #2754
• ucsc-twobittofa @mvdbeek Extract_genomic_dna: use newer ucsc-fatotwobit version #2731
• umi_tools (pysam ImportError: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory) @mvdbeek Add sed requirement, bump samtools requirement for umi_tools #2729 and container rebuild
  • umi_tools_group
  • umi_tools_dedup
• scanpy: @mvdbeek Fix scanpy by using profile #2704
  • scanpy_filter
  • scanpy_normalize
  • scanpy_cluster_reduce_dimension
  • scanpy_plot
  • scanpy_remove_confounders
  • scanpy_inspect
• ncbi_eutils_epost
• quast (UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 14588: ordinal not in range(128)) Use extended base for quast bioconda/bioconda-recipes#19062
• anndata_inspect @mvdbeek needs extended-base container
• snpEff_get_chr_names
• art: @mvdbeek Update art version #2732
  • art_454
  • art_illumina
  • art_solid
• ebi_search_rest_results (HTTP Error 400: )
• ebi_metagenomics_run_downloader (ModuleNotFoundError: No module named 'xmltramp2')
• scpipe
• EMBOSS:_wobble108 (missing perl) @bernt-matthias emboss: fix perl usage and add requirement #2725
• mykrobe_predict
• raceid_filtnormconf @mvdbeek
• repeatmasker_wrapper (most likely sed not working on header)
• heinz_visualization Add xorg-libxrender and fonts to Bandage bioconda/bioconda-recipes#19070
• dropletutils
• CoverageReport2 (perl: unable to load shared object '/usr/local/lib/R/library/grDevices/libs//cairo.so': libXrender.so.1: cannot open shared object file: No such file or directory)
• busco (tar + maybe [makeblastdb] /usr/local/bin/makeblastdb: error while loading shared libraries: libbz2.so.1) busco: add tar requirement #2715 blast 2.2.21 -> 2.2.31: and backport changes from latest bioconda/bioconda-recipes#19016
• simple_weather
• ncbi_eutils_efetch
• onto_tk_get_relationship_id_vs_relationship_def
• migmap (blast: error while loading shared libraries: libbz2.so.1) Bump migmap build, get bzip2 dependency in container bioconda/bioconda-recipes#19084 Use migmap 1.0.3, fixes missing bzip2.so #2738
• glimmer_acgt_content
• cshl_fastx_nucleotides_distribution
• snippy_core (tar unrecognized option '--strip-components=1') snippy: add tar requirement and use -x #2716
• raceid_inspectclusters @mvdbeek
• mothur_chimera_slayer @nsoranzo / Fix link to remove.otu #2702
• glimmer_extract
• scikit_bio_diversity_beta_diversity
• breseq (tar -z) breseq: add tar requirement #2713
• bwa_mem_index_builder_data_manager @mvdbeek Add python 3.7 dependency to bwa_mem index builder #2724
• cshl_fastq_quality_boxplot
• onto_tk_get_terms_by_relationship_type
• hicexplorer_hicplottads
• enasearch_retrieve_analysis_report
• codeml (mv: can't remove '/tmp/tmpx34zwk08/job_working_directory/000/3/tmp5l4xkv9b': Read-only file system)
• ncbi_eutils_einfo
• query_tabular (diff probably caused somewhere due to floating point precision)
• fasttree (docker: invalid reference format: repository name must be lowercase.) @mvdbeek
• gemini_burden
• poretools_squiggle
• pipelign
• stacks2 @bernt-matthias (diff with test data .. don't understand yet)
  • stacks2_ustacks
  • stacks2_denovomap
• stacks @bernt-matthias (all seem to be installation problems: command not found)
  • stacks_cstacks
  • stacks_clonefilter
  • stacks_ustacks
  • stacks_rxstacks
  • stacks_populations
  • stacks_denovomap
  • stacks_genotypes
  • stacks_sstacks
  • stacks_pstacks
  • stacks_refmap
  • stacks_assembleperead
  • stacks_procrad
• samtools @bernt-matthias (does not find test data for cached case because test-data/cached_locally is a symlink) samtools: remove unnecessary symlink in global test-data #2712
  • samtools_bam_to_cram
  • samtools_cram_to_bam
  • samtools_view
• dada2 @bernt-matthias (all but fetcher are due to diff in produced Rdata files, fetcher real bug) dada2 fixes #2705
  • dada2_mergePairs
  • dada2_removeBimeraDenovo
  • dada2_dada
  • dada2_learnErrors
  • dada2_fetcher
• maxbin2 @bernt-matthias (tar wo -z option) maxbin2: add idba out/err only if present (and update bioconda recipe) #2706
• mummer_mummerplot
• Extract_genomic_DNA_1 (faToTwoBit: error while loading shared libraries: libstdc++.so.6:) @mvdbeek Extract_genomic_dna: use newer ucsc-fatotwobit version #2731
• transtermhp (env variable from /etc/conda/activate.d/ not set) transtermhp: add fix for env variable not set in containers #2726
• snap_training
• raceid_clustering @mvdbeek
• glimmer_build_icm
• raxml (python: command not found) raxml: add python requirement #2707
• gemini_annotate
• ruvseq (symlinked R script not found)
• sina
• meme_fimo - @gregvonkuster Update meme and fix tests #2744
• fermi2
• raceid_inspecttrajectory @mvdbeek
• enasearch_retrieve_taxons
• miniasm (error while loading shared libraries: libz.so.1)
• star_fusion
• hicexplorer_hicpca
• glimmer_not_knowledge_based
• resize_coordinate_window @gregvonkuster Fix resize_coordinate_window for python 3 #2745
• newick_display (convert: command not found) newick display: add imagemagick dependency #2710
• poretools_events
• poretools_yield_plot
• trinotate
• taxonomy_krona_chart (/usr/bin/perl: bad interpreter: No such file or directory) update krona to 2.7.1 #2709
• pathview (png output size diff)
• bcftools
  • bcftools_stats (env: can't execute 'perl': No such file or directory)
  • bcftools_plugin_color_chrs (env: can't execute 'perl': No such file or directory)
  • bcftools_plugin_counts (python: command not found)
• bctools
  • bctools_merge_pcr_duplicates (sort: unrecognized option '--compress-program=gzip')
  • bctools_remove_spurious_events (probably missing perl or wrong sort usage)
• graphlan (conda conflict)
  • graphlan (graphlan.py: command not found)
  • graphlan_annotate.py: command not found
• graphlan_annotate
• cshl_fastx_barcode_splitter
• poretools_occupancy
• phyml
• enasearch_search_data
• bowtie2_index_builder_data_manager @mvdbeek Add python 3.7 to bowtie2, fix test data order #2723
• varscan_somatic
• bracken_build_database
• variant_analyzer @bernt-matthias
  • read2mut (matplotlib)
  • mut2read
  • mut2sscs
• jbrowse @abretaud
• abyss-pe @mvdbeek abyss: Add util-linux for column command bioconda/bioconda-recipes#19060
• prinseq (/usr/local/bin/perl shebang + recipe bump was forgotten) prinseq: bump build bioconda/bioconda-recipes#19014
• kraken kraken: test fixes #2721
  • kraken (some docker problem)
  • kraken-report
  • kraken-translate
  • kraken-mpa-report
• kraken2_build_database (the kraken perl script uses @ in path Possible unintended interpolation of @2 in string at /home/berntm/miniconda3/envs/__kraken2@2.0.8_beta/bin/kraken2, needs conda fix)
• valet
• meme_meme - @gregvonkuster - Update meme and fix tests #2744
• pe_histogram (fonts: libfreetype.so.6: cannot open shared object file, JAVA) pe_histogram: add freetype requirement #2728
• raceid_trajectory @mvdbeek
• vcffilter2
• column_order_header_sort
• Bandage: @mvdbeek Fix containerized Bandage #2733
  • bandage_info
  • bandage_image
• structure
• glimmer_long_orfs
• humann2
• rcorrector
• onto_tk_get_relationship_id_vs_relationship_name
• enasearch_retrieve_run_report
• feelnc @abretaud Fix feelnc and orthofinder with singularity #2757
• prokka (version mismatch in test data and maybe tbl2asn to old?) prokka: more flexible testing #2717 still needs docker image rebuild
• snpEff_build_gb
• slamdunk
• hicexplorer_hicplotmatrix
• kraken-filter
• orthofinder_onlygroups @abretaud Fix feelnc and orthofinder with singularity #2757
• prestor_abseq3
• augustus_training @abretaud Augustus 3.3.3 #2759
• kissplice (error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory)
• onto_tk_get_relationship_types
• poretools_qualpos
• goenrichment (fonts, Java) goenrichment: add fonts requirement bioconda/bioconda-recipes#19043
• tb_profiler_profile
• kobas_annotate
• circos (tar -z AND mv: can't remove '/tmp/tmpzvg40cuz/job_working_directory/000/7/tmparfo8udy': Read-only file system) circos: add tar requirement #2714
  • circos
  • circos_tableviewer
• bax2bam (mv: can't remove '/tmp/tmpzvg40cuz/job_working_directory/000/2/tmpnuxq8_9j': Read-only file system)
• htseq_count
• gfa_to_fa
• hmmer3 hmmer3: update and nhmmer test fixes #2720
  • hmmer_nhmmer
  • hmmer_nhmmscan
• mykrobe_genotype
• idr
• masigpro masigpro: add coreutils requirement (for paste) #2722
• poretools_hist
• cshl_fastx_trimmer
• qualimap (Java + fonts) qualimap add fonts requirement bioconda/bioconda-recipes#19041
  • qualimap_multi_bamqc
  • qualimap_rnaseq
• snpEff_download
• glimmer_knowledge_based
• tag_pileup_frequency - tag_pileup_frequency test fixes #2727 - current failure: Applying linter tool_xsd... FAIL, The attribute 'designation' is not allowed
• qiime (uses python2.7 but errors on 'getpwuid(): uid not found: 1001') @mvdbeek needs writable home dir, done. Also needs extended base image, need to rewrite mulled-build
  • qiime_summarize_taxa_through_plots
  • qiime_alpha_rarefaction
  • qiime_upgma_cluster
  • qiime_extract_barcodes
  • qiime_alpha_diversity
  • qiime_compare_categories
  • qiime_split_libraries_fastq
  • qiime_beta_diversity
  • qiime_plot_taxa_summary
  • qiime_jackknifed_beta_diversity
  • qiime_pick_otus
  • qiime_summarize_taxa
  • qiime_validate_mapping_file
  • qiime_count_seqs
  • qiime_filter_otus_from_otu_table
  • qiime_filter_taxa_from_otu_table
  • qiime_pick_open_reference_otus
  • qiime_collapse_samples
  • qiime_beta_diversity_through_plots
  • qiime_multiple_split_libraries_fastq
  • qiime_filter_fasta
  • qiime_pick_closed_reference_otus
  • qiime_split_libraries
  • qiime_filter_samples_from_otu_table
  • qiime_make_emperor
  • qiime_assign_taxonomy
  • qiime_filter_alignment
  • qiime_make_otu_table
  • qiime_core_diversity
  • qiime_make_otu_heatmap
  • qiime_pick_rep_set
  • qiime_multiple_join_paired_ends
• fsd (ModuleNotFoundError: No module named 'matplotlib') fsd+repmatch_gff3: use recent matplotlib #2718
  • fsd_regions (same)
  • sdsd_beforevsafter
  • td
• repmatch_gff3 (ModuleNotFoundError: No module named 'matplotlib') fsd+repmatch_gff3: use recent matplotlib #2718

If you're currently working on one of these tools write your name under the tool, so that we avoid duplicating our work

[wm75]

What about all these errors like:
tool_script.sh: line 9: /tmp/tmpb9_yhhm8/job_working_directory/000/26/COMMAND_VERSION: Read-only file system
Looks like a lot of the above tools are failing with this.

[mvdbeek]

That's something we're working on (the COMMAND_VERSION file doesn't get written in the correct location), but shouldn't be the cause of the error.
If you look for instance at the htseq_count error you have:

/tmp/tmpsv44thj7/job_working_directory/000/19/tool_script.sh: line 9: /tmp/tmpsv44thj7/job_working_directory/000/19/COMMAND_VERSION: Read-only file system
  [Errno 2] No such file or directory: '/tmp/tmpsv44thj7/tmp/26_tmp.sam'
  [Exception type: FileNotFoundError, raised in count.py:52]
[E::hts_open_format] Failed to open file /tmp/tmpsv44thj7/tmp/26_tmp.sam
samtools view: failed to open "/tmp/tmpsv44thj7/tmp/26_tmp.sam" for reading: No such file or directory

So the true error is that something is going wrong with the input sam file.

[mvdbeek]

xref https://github.com/galaxyproject/galaxy/pull/8997/files

[wm75]

Good to know you're fixing this already :-)
Look at the failing raceid tools for examples where I think the command version is the only reason for failure.

[bernt-matthias]

Hi, will try to take care of stacks(2) samtools and maxbin2.

PR: Should be done against iuc-repo? Will IUC testing then be in the simple container?

[mvdbeek]

Yeah, let's test against iuc repo, but no, we don't test against the container yet, I'll see what I can do about #2677 (it should be as simple as that, currently working on the Galaxy PR). If you see and understand any of the errors in the test report feel free to starting PR'ing ... it'll probably still take until Monday or Tuesday for getting things merged into 19.09 / planemo / ... given that it's thanksgiving weekend in the US.

[mvdbeek]

@wm75 after fixing COMMAND_VERSION being mounted in ro I see this:

  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/case.py", line 628, in run
    testMethod()
  File "/Users/mvandenb/src/galaxy/test/functional/test_toolbox.py", line 99, in test_tool
    self.do_it(tool_version=tool_version, test_index=test_index)
  File "/Users/mvandenb/src/galaxy/test/functional/test_toolbox.py", line 36, in do_it
    verify_tool(tool_id, self.galaxy_interactor, resource_parameters=resource_parameters, test_index=test_index, tool_version=tool_version, register_job_data=register_job_data)
  File "/Users/mvandenb/src/galaxy/lib/galaxy/tool_util/verify/interactor.py", line 789, in verify_tool
    raise e
  File "/Users/mvandenb/src/galaxy/lib/galaxy/tool_util/verify/interactor.py", line 785, in verify_tool
    job_stdio = _verify_outputs(testdef, test_history, jobs, tool_id, data_list, data_collection_list, galaxy_interactor, quiet=quiet)
  File "/Users/mvandenb/src/galaxy/lib/galaxy/tool_util/verify/interactor.py", line 1006, in _verify_outputs
    raise JobOutputsError(found_exceptions, job_stdio)
Job in error state.
Job in error state.
Job in error state.
Job in error state.
Job in error state.
-------------------- >> begin captured stdout << ---------------------
Problem in history with id 2891970512fa2d5a - summary of history's datasets and jobs below.
--------------------------------------
| 1 - matrix.filter.rdat (HID - NAME) 
| Dataset State:
|  ok
| Dataset Blurb:
|  7.9 MB
| Dataset Info:
|  The uploaded binary file format cannot be determined automatically, please set the file 'Type' manually
| Peek:
|  <table cellspacing="0" cellpadding="3"><tr><td>binary data</td></tr></table>
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 2 - Clustering using RaceID on data 1: PDF Report (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 3 - Clustering using RaceID on data 1: RDS (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 4 - Clustering using RaceID on data 1: Cell-to-Cluster assignments (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 5 - Clustering using RaceID on data 1: Cluster - Genes per Cluster (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 6 - Clustering using RaceID on data 1: Log (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| Job 5729865256bc2525
| State: 
|  error
| Update Time:
|  2019-11-28T14:22:22.804340
| Create Time:
|  2019-11-28T14:22:09.907655
|
--------------------------------------
| Job 2891970512fa2d5a
| State: 
|  ok
| Update Time:
|  2019-11-28T14:22:08.041991
| Create Time:
|  2019-11-28T14:22:02.955169
|
--------------------------------------
Problem in history with id 2891970512fa2d5a - summary of history's datasets and jobs below.
--------------------------------------
| 1 - matrix.filter.rdat (HID - NAME) 
| Dataset State:
|  ok
| Dataset Blurb:
|  7.9 MB
| Dataset Info:
|  The uploaded binary file format cannot be determined automatically, please set the file 'Type' manually
| Peek:
|  <table cellspacing="0" cellpadding="3"><tr><td>binary data</td></tr></table>
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 2 - Clustering using RaceID on data 1: PDF Report (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 3 - Clustering using RaceID on data 1: RDS (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 4 - Clustering using RaceID on data 1: Cell-to-Cluster assignments (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 5 - Clustering using RaceID on data 1: Cluster - Genes per Cluster (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 6 - Clustering using RaceID on data 1: Log (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| Job 5729865256bc2525
| State: 
|  error
| Update Time:
|  2019-11-28T14:22:22.804340
| Create Time:
|  2019-11-28T14:22:09.907655
|
--------------------------------------
| Job 2891970512fa2d5a
| State: 
|  ok
| Update Time:
|  2019-11-28T14:22:08.041991
| Create Time:
|  2019-11-28T14:22:02.955169
|
--------------------------------------
Problem in history with id 2891970512fa2d5a - summary of history's datasets and jobs below.
--------------------------------------
| 1 - matrix.filter.rdat (HID - NAME) 
| Dataset State:
|  ok
| Dataset Blurb:
|  7.9 MB
| Dataset Info:
|  The uploaded binary file format cannot be determined automatically, please set the file 'Type' manually
| Peek:
|  <table cellspacing="0" cellpadding="3"><tr><td>binary data</td></tr></table>
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 2 - Clustering using RaceID on data 1: PDF Report (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 3 - Clustering using RaceID on data 1: RDS (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 4 - Clustering using RaceID on data 1: Cell-to-Cluster assignments (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 5 - Clustering using RaceID on data 1: Cluster - Genes per Cluster (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 6 - Clustering using RaceID on data 1: Log (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| Job 5729865256bc2525
| State: 
|  error
| Update Time:
|  2019-11-28T14:22:22.804340
| Create Time:
|  2019-11-28T14:22:09.907655
|
--------------------------------------
| Job 2891970512fa2d5a
| State: 
|  ok
| Update Time:
|  2019-11-28T14:22:08.041991
| Create Time:
|  2019-11-28T14:22:02.955169
|
--------------------------------------
Problem in history with id 2891970512fa2d5a - summary of history's datasets and jobs below.
--------------------------------------
| 1 - matrix.filter.rdat (HID - NAME) 
| Dataset State:
|  ok
| Dataset Blurb:
|  7.9 MB
| Dataset Info:
|  The uploaded binary file format cannot be determined automatically, please set the file 'Type' manually
| Peek:
|  <table cellspacing="0" cellpadding="3"><tr><td>binary data</td></tr></table>
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 2 - Clustering using RaceID on data 1: PDF Report (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 3 - Clustering using RaceID on data 1: RDS (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 4 - Clustering using RaceID on data 1: Cell-to-Cluster assignments (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 5 - Clustering using RaceID on data 1: Cluster - Genes per Cluster (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 6 - Clustering using RaceID on data 1: Log (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| Job 5729865256bc2525
| State: 
|  error
| Update Time:
|  2019-11-28T14:22:22.804340
| Create Time:
|  2019-11-28T14:22:09.907655
|
--------------------------------------
| Job 2891970512fa2d5a
| State: 
|  ok
| Update Time:
|  2019-11-28T14:22:08.041991
| Create Time:
|  2019-11-28T14:22:02.955169
|
--------------------------------------
Problem in history with id 2891970512fa2d5a - summary of history's datasets and jobs below.
--------------------------------------
| 1 - matrix.filter.rdat (HID - NAME) 
| Dataset State:
|  ok
| Dataset Blurb:
|  7.9 MB
| Dataset Info:
|  The uploaded binary file format cannot be determined automatically, please set the file 'Type' manually
| Peek:
|  <table cellspacing="0" cellpadding="3"><tr><td>binary data</td></tr></table>
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 2 - Clustering using RaceID on data 1: PDF Report (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 3 - Clustering using RaceID on data 1: RDS (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 4 - Clustering using RaceID on data 1: Cell-to-Cluster assignments (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 5 - Clustering using RaceID on data 1: Cluster - Genes per Cluster (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| 6 - Clustering using RaceID on data 1: Log (HID - NAME) 
| Dataset State:
|  error
| Dataset Blurb:
|  error
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  *Peek unavilable.*
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*
|
--------------------------------------
| Job 5729865256bc2525
| State: 
|  error
| Update Time:
|  2019-11-28T14:22:22.804340
| Create Time:
|  2019-11-28T14:22:09.907655
|
--------------------------------------
| Job 2891970512fa2d5a
| State: 
|  ok
| Update Time:
|  2019-11-28T14:22:08.041991
| Create Time:
|  2019-11-28T14:22:02.955169
|
--------------------------------------

--------------------- >> end captured stdout << ----------------------

we may need to specify the filetype on upload ?

[bgruening]

One error-class that we should look out for is that all Java based tools that do some plotting might miss some fonts in the container. Keep an eye on this.

[mvdbeek]

Is there some workaround for missing fonts ? A conda package maybe, or a new base container ?

[bgruening]

Yes, but we need to understand the problem at first. For conda-forge we now have: fonts-conda-ecosystem ... but no clue if this fixes our problem. Just something to look out for and ping me if you see it ... goenrichment seems to have this exact problem.

[mvdbeek]

Oh, and I forgot, the tests run on python 3.7, so there may be another class of errors we may hit in the cheetah templates.

[bgruening]

All or nothing! 🎉

[mvdbeek]

For raceid I had to check the actual output file as there is a stderr redirect:

Error in dyn.load(file, DLLpath = DLLpath, ...) :
  unable to load shared object '/usr/local/lib/R/library/rgl/libs/rgl.so':
  libGL.so.1: cannot open shared object file: No such file or directory
Warning: namespace 'RaceID' is not available and has been replaced

That's of course a problem for debugging things remotely.
The issue itself seems difficult / not possible to fix with conda-only means:
conda-forge/pygridgen-feedstock#10 (comment)

We may need to provide an alternative docker base image for these cases

[bernt-matthias]

Added dada2 to my list and documented the problems in the original list. Some seem real bugs (samtools and the dada_fetcher). Maxbin 2 just stumbles over tar -x.

For dada2 its probably because it uses Rdata files which might be compressed differently on the minimal system and therefore it results in a diff.

• Should we just allow larger diff (essentially just test for file presence) or
• disable compression for the Rdata files (would also save computation time but need more memory)

I have no idea about why the old stacks tools do not find any of the binaries. Might this be a (temporary) problem with conda?

For stacks2 there are minor differences in the test data (I will check if we can/should allow them).