Merge branch 'dev' into dedup-gc-overhead-fix

Author: jfy133

.github/CONTRIBUTING.md

@@ -18,8 +18,9 @@ If you'd like to write some code for nf-core/eager, the standard workflow is as
 1. Check that there isn't already an issue about your idea in the [nf-core/eager issues](https://github.com/nf-core/eager/issues) to avoid duplicating work
     * If there isn't one already, please create one so that others know you're working on this
 2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [nf-core/eager repository](https://github.com/nf-core/eager) to your GitHub account
-3. Make the necessary changes / additions within your forked repository
-4. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged
+3. Make the necessary changes / additions within your forked repository (following [code contribution guidelines](https://github.com/nf-core/eager/blob/dev/docs/code_contribution.md))
+4. Use `nf-core schema build .` and add any new parameters to the pipeline JSON schema (requires nf-core tools >= 1.10).
+5. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged
 
 If you're not used to this workflow with git, you can start with some [docs from GitHub](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests) or even their [excellent `git` resources](https://try.github.io/).
 

.github/workflows/ci.yml

@@ -119,6 +119,9 @@ jobs:
       - name: GENOTYPING_PC Test running pileupCaller
         run: |
           nextflow run ${GITHUB_WORKSPACE} -profile test_tsv_humanbam,docker --dedupper 'dedup' --run_genotyping --genotyping_tool 'pileupcaller'
+      - name: GENOTYPING_ANGSD Test running ANGSD genotype likelihood calculation
+        run: |
+          nextflow run ${GITHUB_WORKSPACE} -profile test_tsv_humanbam,docker --dedupper 'dedup' --run_genotyping --genotyping_tool 'angsd'
       - name: SKIPPING Test checking all skip steps work i.e. input bam, skipping straight to genotyping
         run: |
           nextflow run ${GITHUB_WORKSPACE} -profile test_tsv_bam,docker --skip_fastqc --skip_adapterremoval --skip_deduplication --skip_qualimap --skip_preseq --skip_damage_calculation --run_genotyping --genotyping_tool 'freebayes'

CHANGELOG.md

@@ -11,19 +11,20 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 * **Major** Automated Cloud Tests with large-scale data on [AWS](https://aws.amazon.com/)
 * **Major** Re-wrote input logic to accept a TSV 'map' file in addition to direct paths to FASTQ
+* **Major** Added JSON Schema, enabling web GUI for configuration of pipeline
 * **Major** Lane and library merging implement
   * When using TSV input, one libraries with the multiple _lane_ will be merged together, before mapping
     * Strip FASTQ will also produce a lane merged 'raw' but 'stripped' FASTQ file
   * When using TSV input, one sample with multiple (same treatment) libraries will be merged together.
   * Important: direct FASTQ paths will not have this functionality. TSV is required.
 * [#40](https://github.com/nf-core/eager/issues/40) - Added the pileupCaller genotyper from [sequenceTools](https://github.com/stschiff/sequenceTools)
 * Added sanity check and clearer error message when `--fasta_index` is provided and filepath does not end in `.fai`.
-* Added basic json_schema
 * Improved error messages
 * Added ability for automated emails using `mailutils` to also send MultiQC reports
 * General documentation additions and cleaning, updated figures with CC-BY license
 * Added large 'fullsize' dataset test-profiles for ancient fish, human, and a draft pathogen contexts.
 * [#257](https://github.com/nf-core/eager/issues/257) Added the bowtie2 aligner as option for mapping, following Poullet and Orlando 2020 doi: [10.3389/fevo.2020.00105](https://doi.org/10.3389/fevo.2020.00105)
+* [#451] Adds ANGSD genotype likelihood calculations as alternative to typical 'genotypers'
 
 ### `Fixed`
 
@@ -52,7 +53,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 * Latest version of Kraken2 (2.0.9beta)
 * Latest version of FreeBayes (1.3.2)
 * Latest version of xopen (0.9.0)
-* *Adding* in Bowtie 2 (2.4.1)
+* Added Bowtie 2 (2.4.1)
 
 ## [2.1.0] - 2020-03-05 - "Ravensburg"
 

README.md

@@ -55,11 +55,13 @@ Additional functionality contained by the pipeline currently includes:
 
 * Damage removal/clipping for UDG+/UDG-half treatment protocols (`BamUtil`)
 * Damaged reads extraction and assessment (`PMDTools`)
+* Nuclear DNA contamination estimation of human samples (`angsd`)
 
 #### Genotyping
 
 * Creation of VCF genotyping files (`GATK UnifiedGenotyper`, `GATK HaplotypeCaller` and `FreeBayes`)
 * Creation of EIGENSTRAT genotyping files (`pileupCaller`)
+* Creation of Genotype Likelihood files (`angsd`)
 * Consensus sequence FASTA creation (`VCF2Genome`)
 * SNP Table generation (`MultiVCFAnalyzer`)
 
@@ -88,7 +90,7 @@ Additional functionality contained by the pipeline currently includes:
 
         nextflow run nf-core/eager -profile <docker/singularity/conda>,test
 
-5. Start running your own ancient DNA analysis!
+5. Start running your own ancient DNA analysis either via the command-line or [GUI](https://nf-co.re/launch)! Command line example:
 
         nextflow run nf-core/eager -profile <docker/singularity/conda> --input '*_R{1,2}.fastq.gz' --fasta '<your_reference>.fasta'
 
@@ -105,20 +107,20 @@ as described in the documentation.
 
 The nf-core/eager pipeline comes with documentation about the pipeline, found in the `docs/` directory or on the main homepage of the nf-core project:
 
-1. [Nextflow Installation](https://nf-co.re/usage/installation)
+1. [Nextflow installation](https://nf-co.re/usage/installation)
 2. Pipeline configuration
     * [Pipeline installation](https://nf-co.re/usage/local_installation)
     * [Adding your own system config](https://nf-co.re/usage/adding_own_config)
     * [Reference genomes](https://nf-co.re/usage/reference_genomes)
 3. [Running the pipeline](docs/usage.md)
 4. [Output and how to interpret the results](docs/output.md)
-5. [EAGER2 Code Contribution Guidelines](code_contribution.md)
-6. [nf-core/nextflow Troubleshooting](https://nf-co.re/usage/troubleshooting)
-7. [EAGER Troubleshooting](docs/troubleshooting.md)
+5. [nf-core/eager code contribution checklist and guidelines](docs/code_contribution.md)
+6. [nf-core and Nextflow troubleshooting](https://nf-co.re/usage/troubleshooting)
+7. [nf-core/eager troubleshooting](docs/troubleshooting.md)
 
 ## Credits
 
-This pipeline was mostly written by Alexander Peltzer ([apeltzer](https://github.com/apeltzer)), with contributions from [Stephen Clayton](https://github.com/sc13-bioinf), [James A. Fellows Yates](https://github.com/jfy133), [Thiseas C. Lamnidis](https://github.com/TCLamnidis), [Maxime Borry](https://github.com/maxibor), [Zandra Fagernäs](https://github.com/ZandraFagernas), [Aida Andrades Valtueña](https://github.com/aidaanva) and [Maxime Garcia](https://github.com/MaxUlysse). If you want to contribute, please open an issue (or even better, a pull request!) and ask to be added to the project - everyone is welcome to contribute here!
+This pipeline was mostly written by Alexander Peltzer ([apeltzer](https://github.com/apeltzer)) and [James A. Fellows Yates](https://github.com/jfy133), with contributions from [Stephen Clayton](https://github.com/sc13-bioinf), [Thiseas C. Lamnidis](https://github.com/TCLamnidis), [Maxime Borry](https://github.com/maxibor), [Zandra Fagernäs](https://github.com/ZandraFagernas), [Aida Andrades Valtueña](https://github.com/aidaanva) and [Maxime Garcia](https://github.com/MaxUlysse). If you want to contribute, please open an issue (or even better, a pull request!) and ask to be added to the project - everyone is welcome to contribute here!
 
 ## Authors (alphabetical)
 

assets/multiqc_config.yaml

@@ -88,10 +88,10 @@ top_modules:
             - '*_postfilterflagstat.stats'
     - 'dedup'
     - 'picard'
+    - 'preseq'
     - 'damageprofiler'
-    - 'qualimap'
     - 'mtnucratio'
-    - 'preseq'
+    - 'qualimap'
     - 'sexdeterrmine'
     - 'gatk'
     - 'multivcfanalyzer':
@@ -151,13 +151,13 @@ table_columns_visible:
         3 Prime2: False
         mean_readlength: True
         median: True
+    mtnucratio: 
+        mt_nuc_ratio: True
     QualiMap:
         mean_coverage: True
         1_x_pc: True
         5_x_pc: True
         percentage_aligned: False
-    mtnucratio: 
-        mt_nuc_ratio: True
     MultiVCFAnalyzer:
         Heterozygous SNP alleles (percent): True
 
@@ -205,6 +205,10 @@ table_columns_placement:
         3 Prime2: 730
         mean_readlength: 740
         median: 750
+    mtnucratio:
+        mtreads: 760
+        mt_cov_avg: 770
+        mt_nuc_ratio: 780
     QualiMap:
         mean_coverage: 800
         median_coverage: 810
@@ -214,10 +218,6 @@ table_columns_placement:
         4_x_pc: 850
         5_x_pc: 860
         avg_gc: 870
-    mtnucratio:
-        mtreads: 900
-        mt_cov_avg: 910
-        mt_nuc_ratio: 920
     sexdeterrmine:
         RateX: 100
         RateY: 1010

conf/base.config

@@ -76,6 +76,10 @@ process {
     errorStrategy = 'ignore'
   }
 
+  withName:damageprofiler {
+    errorStrategy = { task.exitStatus in [1,143,137,104,134,139] ? 'retry' : 'finish' }
+  }
+
   // Add 141 ignore due to unclean pipe closing by pmdtools https://github.com/pontussk/PMDtools/issues/7
   withName: pmdtools {
     errorStrategy = { task.exitStatus in [141] ? 'ignore' : 'retry' }

docs/code_contribution.md

@@ -11,11 +11,14 @@ The typical workflow for adding a new module is as follows:
 3. Define the output channel if needed (see below).
 4. Add any new flags/options to `nextflow.config` with a default (see below).
 5. Add any new flags/options to the help message (for integer/text parameters, print to help the corresponding `nextflow.config` parameter).
-6. Add any new software to the `scrape_software_versions.py` secript in `bin/` and the version command to the `scrape_software_versions` process in `main.nf`.
-7. Do a local test that the new code works properly
-8. Add a new test command in `.github/workflow/ci.yaml`.
-9. Add new flags/options to 'usage' documentation under `docs/usage.md`.
-10. Add any descriptions of MultiQC report sections and output files to `docs/output.md`.
+6. Add sanity checks for all relevant parameters.
+7. Add any new software to the `scrape_software_versions.py` script in `bin/` and the version command to the `scrape_software_versions` process in `main.nf`.
+8. Do local tests that the new code works properly and as expected.
+9. Add a new test command in `.github/workflow/ci.yaml`.
+10. If applicable add a [MultiQC](https://https://multiqc.info/) module.
+11. Update MultiQC config `assests/multiqc_config.yaml` so relevant suffixes, name clean up, General Statistics Table column order, and module figures are in the right order.
+12. Add new flags/options to 'usage' documentation under `docs/usage.md`.
+13. Add any descriptions of MultiQC report sections and output files to `docs/output.md`.
 
 ## Default Values