Add TSV input funcionality and extra goodies.

.github/workflows/awstest.yml

@@ -0,0 +1,30 @@
+name: nf-core AWS test
+# This workflow is triggered on push to the master branch and on published releases.
+# It runs the -profile 'test' on AWS batch
+
+on:
+  push:
+    branches:
+      - master
+  release:
+    types: [published]
+
+jobs:
+  run-awstest:
+    name: Run AWS test
+    runs-on: ubuntu-latest
+    steps:
+      - name: Setup Miniconda
+        uses: goanpeca/setup-miniconda@v1.0.2
+        with:
+          auto-update-conda: true
+          python-version: 3.7
+      - name: Install awscli
+        run: conda install -c conda-forge awscli
+      - name: Start AWS batch job
+        env:
+          AWS_ACCESS_KEY_ID: ${{secrets.AWS_KEY_ID}}
+          AWS_SECRET_ACCESS_KEY: ${{secrets.AWS_KEY_SECRET}}
+          TOWER_ACCESS_TOKEN: ${{secrets.TOWER_ACCESS_TOKEN}}
+        run: |
+          aws batch submit-job --region eu-west-1 --job-name nf-core-eager --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' --job-definition nextflow --container-overrides '{"command": ["nf-core/eager", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/eager/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/eager/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' 

.gitignore

@@ -6,3 +6,4 @@ results/
 tests/
 testing/
 *.pyc
+main_playground.nf

CHANGELOG.md

@@ -9,20 +9,35 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ### `Added`
 
+* **Major** Automated Cloud Tests with large-scale data on [AWS](https://aws.amazon.com/)
+* **Major** Re-wrote input logic to accept a TSV 'map' file in addition to direct paths to FASTQ
+* **Major** Lane and library merging implement
+  * When using TSV input, one libraries with the multiple _lane_ will be merged together, before mapping
+    * Strip FASTQ will also produce a lane merged 'raw' but 'stripped' FASTQ file
+  * When using TSV input, one sample with multiple (same treatment) libraries will be merged together.
+  * Important: direct FASTQ paths will not have this functionality. TSV is required.
 * Added sanity check and clearer error message when `--fasta_index` is provided and filepath does not end in `.fai`.
+* Added basic json_schema
+* Improved error messages
+* Added ability for automated emails using `mailutils` to also send MultiQC reports
+* General documentation additions and cleaning, updated figures with CC-BY license
 
 ### `Fixed`
 
 * [#368](https://github.com/nf-core/eager/issues/368) - Fixed the profile `test` to contain a parameter for `--paired_end`.
 * Mini bugfix for typo in line 1260+1261
-* Added basic json_schema
+* [#374](https://github.com/nf-core/eager/issues/374) - Fixed output documentation rendering not containing images
 * [#379](https://github.com/nf-core/eager/issues/378) - Fixed insufficient memory requirements for FASTQC edge case
-* Set correct recommended bwa mapping parameters from [Schubert et al. 2012](https://doi.org/10.1186/1471-2164-13-178)
+* [#390](https://github.com/nf-core/eager/issues/390) - Renamed clipped/merged output directory to be more descriptive
+* [#398](https://github.com/nf-core/eager/issues/498) - Stopped incompatible FASTA indexes being accepted
+* [#400](https://github.com/nf-core/eager/issues/400) -  Set correct recommended bwa mapping parameters from [Schubert et al. 2012](https://doi.org/10.1186/1471-2164-13-178)
+* [#410](https://github.com/nf-core/eager/issues/410) - Fixed nf-core/configs not being loaded properly
+* Increase MultiQC process memory requirements to ensure enough memory for large runs
 
 ### `Dependencies`
 
 * Latest version of DeDup (0.12.6) which now reports mapped reads after deduplication
-* Latest version of ANGSD (0.933) which doesn't seg fault when running contamination on BAMs with insufficent reads
+* Latest version of ANGSD (0.933) which doesn't seg fault when running contamination on BAMs with insufficient reads
 
 ## [2.1.0] - 2020-03-05 - "Ravensburg"
 

README.md

@@ -85,7 +85,7 @@ Additional functionality contained by the pipeline currently includes:
 
 5. Start running your own ancient DNA analysis!
 
-        nextflow run nf-core/eager -profile <docker/singularity/conda> --reads '*_R{1,2}.fastq.gz' --fasta '<your_reference>.fasta'
+        nextflow run nf-core/eager -profile <docker/singularity/conda> --input '*_R{1,2}.fastq.gz' --fasta '<your_reference>.fasta'
 
 6. Once your run has completed successfully, clean up the intermediate files.
 
@@ -136,11 +136,12 @@ Those who have provided conceptual guidance, suggestions, bug reports etc.
 * [Irina Velsko](https://github.com/ivelsko)
 * [Katerine Eaton](https://github.com/ktmeaton)
 * [Luc Venturini](https://github.com/lucventurini)
-* Marcel Keller
+* [Marcel Keller](https://github.com/marcel-keller)
 * [Pierre Lindenbaum](https://github.com/lindenb)
 * [Pontus Skoglund](https://github.com/pontussk)
 * [Raphael Eisenhofer](https://github.com/EisenRa)
 * [Torsten Günter](https://bitbucket.org/tguenther/)
+* [Kevin Lord](https://github.com/lordkev)
 
 If you've contributed and you're missing in here, please let us know and we will add you in of course!
 

assets/dummy_postfilterflagstat.stats

@@ -0,0 +1 @@
+

assets/multiqc_config.yaml

@@ -1,10 +1,36 @@
+custom_logo: 'nf-core_eager_logo.png'
 custom_logo_url: https://github.com/nf-core/eager/
 custom_logo_title: 'nf-core/eager'
 
 report_comment: >
     This report has been generated by the <a href="https://github.com/nf-core/eager" target="_blank">nf-core/eager</a>
     analysis pipeline. For information about how to interpret these results, please see the
     <a href="https://github.com/nf-core/eager" target="_blank">documentation</a>.
+extra_fn_clean_exts:
+    - '.fastq.gz'
+    - '.fastq'
+    - '.fq.gz'
+    - '.fq'
+    - '.bam'
+    - '_fastp'
+    - '.pe.settings'
+    - '.se.settings'
+    - '.settings'
+    - '.pe.combined'
+    - '.se.truncated'
+    - '.mapped'
+    - '.mapped_rmdup'
+    - '.mapped_rmdup_stats'
+    - '_libmerged_rg_rmdup'
+    - '_libmerged_rg_rmdup_stats'
+    - '_postfilterflagstat.stats'
+    - '_flagstat.stat'
+    - '.filtered'
+    - '.filtered_rmdup'
+    - '.filtered_rmdup_stats'
+    - '_libmerged_rg_add_stats'
+    - '_rmdup'
+
 top_modules:
      - 'fastqc':
         name: 'FastQC (pre-AdapterRemoval)'
@@ -62,20 +88,21 @@ table_columns_visible:
         mapped_passed: True
     DeDup:
         dup_rate: False
-    QualiMap:
-        mean_coverage: True
-        1_x_pc: True
-        5_x_pc: True
-        percentage_aligned: False
-    mtnucratio: 
-        mt_nuc_ratio: True
     DamageProfiler:
         3 Prime1: True
         3 Prime2: True
         5 Prime1: True
         5 Prime2: True
         mean_readlength: True
         median_readlength: True
+    QualiMap:
+        mean_coverage: True
+        1_x_pc: True
+        5_x_pc: True
+        percentage_aligned: False
+    mtnucratio: 
+        mt_nuc_ratio: True
+
 table_columns_placement:
     FastQC (pre-AdapterRemoval):
         total_sequences: 100
@@ -96,6 +123,13 @@ table_columns_placement:
         endogenous_dna_post: 590
     DeDup:
         clusterfactor: 600
+    DamageProfiler:
+        3 Prime1: 1000
+        3 Prime2: 1010
+        5 Prime1: 1020
+        5 Prime2: 1030
+        mean_readlength: 1040
+        median_readlength: 1050
     QualiMap:
         mean_coverage: 700
         median_coverage: 710
@@ -112,13 +146,6 @@ table_columns_placement:
     sexdeterrmine:
         RateX: 900
         RateY: 910
-    DamageProfiler:
-        3 Prime1: 1000
-        3 Prime2: 1010
-        5 Prime1: 1020
-        5 Prime2: 1030
-        mean_readlength: 1040
-        median_readlength: 1050
 read_count_multiplier: 1
 read_count_prefix: ''
 read_count_desc: ''

conf/test_tsv.config

@@ -0,0 +1,22 @@
+/*
+ * -------------------------------------------------
+ *  Nextflow config file for running tests
+ * -------------------------------------------------
+ * Defines bundled input files and everything required
+ * to run a fast and simple test. Use as follows:
+ * nextflow run nf-core/eager -profile test, docker (or singularity, or conda)
+ */
+
+params {
+  config_profile_name = 'Test profile'
+  config_profile_description = 'Minimal test dataset to check pipeline function'
+  // Limit resources so that this can run on GitHub Actions
+  max_cpus = 2
+  max_memory = 6.GB
+  max_time = 48.h
+  genome = false
+  //Input data
+  input = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/testdata/Mammoth/mammoth_design_fastq.tsv'
+  // Genome references
+  fasta = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/reference/Mammoth/Mammoth_MT_Krause.fasta'
+}

conf/test_tsv_bam.config

@@ -0,0 +1,22 @@
+/*
+ * -------------------------------------------------
+ *  Nextflow config file for running tests
+ * -------------------------------------------------
+ * Defines bundled input files and everything required
+ * to run a fast and simple test. Use as follows:
+ * nextflow run nf-core/eager -profile test, docker (or singularity, or conda)
+ */
+
+params {
+  config_profile_name = 'Test profile'
+  config_profile_description = 'Minimal test dataset to check pipeline function'
+  // Limit resources so that this can run on Travis
+  max_cpus = 2
+  max_memory = 6.GB
+  max_time = 48.h
+  genome = false
+  //Input data
+  input = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/testdata/Mammoth/mammoth_design_bam.tsv'
+  // Genome references
+  fasta = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/reference/Mammoth/Mammoth_MT_Krause.fasta'
+}

conf/test_tsv_complex.config

@@ -0,0 +1,22 @@
+/*
+ * -------------------------------------------------
+ *  Nextflow config file for running tests
+ * -------------------------------------------------
+ * Defines bundled input files and everything required
+ * to run a fast and simple test. Use as follows:
+ * nextflow run nf-core/eager -profile test, docker (or singularity, or conda)
+ */
+
+params {
+  config_profile_name = 'Test profile'
+  config_profile_description = 'Minimal test dataset to check pipeline function'
+  // Limit resources so that this can run on GitHub Actions
+  max_cpus = 2
+  max_memory = 6.GB
+  max_time = 48.h
+  genome = false
+  //Input data
+  input = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/testdata/Mammoth/mammoth_design_fastq_multilane_multilib.tsv'
+  // Genome references
+  fasta = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/reference/Mammoth/Mammoth_MT_Krause.fasta'
+}

conf/test_tsv_fna.config

@@ -0,0 +1,22 @@
+/*
+ * -------------------------------------------------
+ *  Nextflow config file for running tests
+ * -------------------------------------------------
+ * Defines bundled input files and everything required
+ * to run a fast and simple test. Use as follows:
+ * nextflow run nf-core/eager -profile test, docker (or singularity, or conda)
+ */
+
+params {
+  config_profile_name = 'Test profile'
+  config_profile_description = 'Minimal test dataset to check pipeline function'
+  // Limit resources so that this can run on Travis
+  max_cpus = 2
+  max_memory = 6.GB
+  max_time = 48.h
+  genome = false
+  //Input data
+  input = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/testdata/Mammoth/mammoth_design_fastq.tsv'
+  // Genome references
+  fasta = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/reference/Mammoth/Mammoth_MT_Krause.fna'
+}

conf/test_tsv_humanbam.config

@@ -0,0 +1,23 @@
+/*
+ * -------------------------------------------------
+ *  Nextflow config file for running tests
+ * -------------------------------------------------
+ * Defines bundled input files and everything required
+ * to run a fast and simple test. Use as follows:
+ * nextflow run nf-core/eager -profile test, docker (or singularity, or conda)
+ */
+
+params {
+  config_profile_name = 'Test profile'
+  config_profile_description = 'Minimal test dataset to check pipeline function'
+  // Limit resources so that this can run on Travis
+  max_cpus = 2
+  max_memory = 6.GB
+  max_time = 48.h
+  genome = false
+  //Input data
+  input = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/testdata/Human/human_design_bam.tsv'
+  // Genome references
+  fasta = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/reference/Mammoth/Mammoth_MT_Krause.fasta'
+  sexdeterrmine_bedfile = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/reference/Human/1240K.pos.list_hs37d5.0based.bed.gz'
+}

conf/test_tsv_kraken.config

@@ -0,0 +1,25 @@
+/*
+ * -------------------------------------------------
+ *  Nextflow config file for running tests
+ * -------------------------------------------------
+ * Defines bundled input files and everything required
+ * to run a fast and simple test. Use as follows:
+ * nextflow run nf-core/eager -profile test, docker (or singularity, or conda)
+ */
+
+params {
+  config_profile_name = 'Test profile kraken'
+  config_profile_description = 'Minimal test dataset to check pipeline function with kraken metagenomic profiler'
+  // Limit resources so that this can run on Travis
+  max_cpus = 2
+  max_memory = 6.GB
+  max_time = 48.h
+  genome = false
+  //Input data
+  metagenomic_tool = 'kraken'
+  run_metagenomic_screening = true
+  input = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/testdata/Mammoth/mammoth_design_fastq.tsv'
+  // Genome references
+  fasta = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/reference/Mammoth/Mammoth_MT_Krause.fasta'
+  database = 'https://github.com/nf-core/test-datasets/raw/eager/databases/kraken/eager_test.tar.gz'
+}

conf/test_tsv_pretrim.config

@@ -0,0 +1,22 @@
+/*
+ * -------------------------------------------------
+ *  Nextflow config file for running tests
+ * -------------------------------------------------
+ * Defines bundled input files and everything required
+ * to run a fast and simple test. Use as follows:
+ * nextflow run nf-core/eager -profile test, docker (or singularity, or conda)
+ */
+
+params {
+  config_profile_name = 'Test profile'
+  config_profile_description = 'Minimal test dataset to check pipeline function'
+  // Limit resources so that this can run on Travis
+  max_cpus = 2
+  max_memory = 6.GB
+  max_time = 48.h
+  genome = false
+  //Input data
+  input = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/testdata/Mammoth/mammoth_design_fastq_pretrim.tsv'
+  // Genome references
+  fasta = 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/reference/Mammoth/Mammoth_MT_Krause.fasta'
+}

docs/faq.md

@@ -53,6 +53,25 @@ the output folder. Otherwise you may end up a lot of large intermediate files
 being left! You can clean a nextflow run of all intermediate files with
 `nextflow clean -f -k` or delete the `work/` directory.
 
+## I get a file name collision error during library merging
+
+When using TSV input, nf-core EAGER will attempt to merge all files with the
+same `Sample ID`. However, if you have specified the same `Library_ID` for two
+sets of FASTQ files you will likely receive an error such as
+
+```bash
+Error executing process > 'library_merge (JK2782)'
+Caused by:
+  Process `library_merge` input file name collision -- There are multiple input files for each of the following file names: JK2782.mapped_rmdup.bam.csi, JK2782.mapped_rmdup.bam
+Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
+Execution cancelled -- Finishing pending tasks before exit
+```
+
+This could happen when you specify the same `Library_ID` but with different
+sequencing configurations (PE vs SE).
+
+In this case you must modify your `Library_ID` accordingly to make them unique.
+
 ## How to investigate a failed run
 
 As with most pipelines, nf-core/eager can sometimes fail either through a