Dev

.github/CONTRIBUTING.md

@@ -55,7 +55,7 @@ These tests are run both with the latest available version of `Nextflow` and als
 
 ## Getting help
 
-For further information/help, please consult the [nf-core/eager documentation](https://nf-co.re/eager/latest/usage) and don't hesitate to get in touch on the nf-core Slack [#eager](https://nfcore.slack.com/channels/eager) channel ([join our Slack here](https://nf-co.re/join/slack)).
+For further information/help, please consult the [nf-core/eager documentation](https://nf-co.re/eager/usage) and don't hesitate to get in touch on the nf-core Slack [#eager](https://nfcore.slack.com/channels/eager) channel ([join our Slack here](https://nf-co.re/join/slack)).
 
 # Code Contribution Guidelines
 

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,10 +1,15 @@
-# nf-core/eager bug report
+---
+name: Bug report
+about: Report something that is broken or incorrect
+labels: bug
+---
 
 <!--
+# nf-core/eager bug report
+
 Hi there!
 
 Thanks for telling us about a problem with the pipeline.
-
 Please delete this text and anything that's not relevant from the template below:
 -->
 
@@ -17,12 +22,9 @@ Have you checked in the following places for your error?:
 - [ ] [Troubleshooting](https://nf-co.re/usage/troubleshooting)
       (for nf-core specific information)
 
-Please also check the the corresponding version's documentation on github, if not
-testing the latest release.
-
-## Describe the bug
+## Description of the bug
 
-A clear and concise description of what the bug is.
+<!-- A clear and concise description of what the bug is. -->
 
 ## Steps to reproduce
 
@@ -33,13 +35,13 @@ Steps to reproduce the behaviour:
 
 ## Expected behaviour
 
-A clear and concise description of what you expected to happen.
+<!-- A clear and concise description of what you expected to happen. -->
 
 ## Log files
 
-1. The command used to run the pipeline
-2. The `.nextflow.log` file (which is a hidden file in whichever place you _ran_
-   the pipeline from - not necessarily in the output directory!)
+1. Command line: <!-- [e.g. `nextflow run ...`] -->
+2. The `.nextflow.log` file (which is a hidden file in whichever place you _ran_ the pipeline from - not necessarily in the output directory!)
+3. See error: <!-- [Please provide your error message] -->
 
 ## System
 
@@ -54,10 +56,10 @@ A clear and concise description of what you expected to happen.
 
 ## Container engine
 
-- Engine: <!-- [e.g. Conda, Docker or Singularity] -->
-- version: <!-- [e.g. 2.1.0] -->
-- Image tag: <!-- [e.g. nfcore/eager:2.1.0] -->
+- Engine: <!-- [e.g. Conda, Docker, Singularity or Podman] -->
+- version: <!-- [e.g. 1.0.0] -->
+- Image tag: <!-- [e.g. nfcore/eager:1.0.0] -->
 
 ## Additional context
 
-Add any other context about the problem here.
+<!-- Add any other context about the problem here. -->

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,8 @@
+blank_issues_enabled: false
+contact_links:
+  - name: Join nf-core
+    url: https://nf-co.re/join
+    about: Please join the nf-core community here
+  - name: "Slack #eager channel"
+    url: https://nfcore.slack.com/channels/eager
+    about: Discussion about the nf-core/eager pipeline

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,5 +1,11 @@
-# nf-core/eager feature request
+---
+name: Feature request
+about: Suggest an idea for the nf-core website
+labels: enhancement
+---
+
 <!--
+# nf-core/eager feature request
 
 Hi there!
 

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,10 +1,14 @@
-# nf-core/eager pull request
 <!--
+# nf-core/eager pull request
 
 Many thanks for contributing to nf-core/eager!
 
 Please fill in the appropriate checklist below (delete whatever is not relevant).
 These are the most common things requested on pull requests (PRs).
+
+Remember that PRs should be made against the dev branch, unless you're preparing a pipeline release.
+
+Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-core/eager/tree/master/.github/CONTRIBUTING.md)
 -->
 
 ## PR checklist

.github/workflows/awsfulltest.yml

@@ -4,8 +4,9 @@ name: nf-core AWS full size tests
 # It runs the -profile 'test_full' on AWS batch
 
 on:
-  release:
-    types: [published]
+  workflow_run:
+    workflows: ["nf-core Docker push (release)"]
+    types: [completed]
   workflow_dispatch:
 
 jobs:
@@ -15,13 +16,14 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Setup Miniconda
-        uses: goanpeca/setup-miniconda@v1.0.2
+        uses: conda-incubator/setup-miniconda@v2
         with:
           auto-update-conda: true
           python-version: 3.7
       - name: Install awscli
         run: conda install -c conda-forge awscli
       - name: Start AWS batch job
+        # TODO nf-core: You can customise AWS full pipeline tests as required
         # Add full size test data (but still relatively small datasets for few samples)
         # on the `test_full.config` test runs with only one set of parameters
         # Then specify `-profile test_full` instead of `-profile test` on the AWS batch command
@@ -38,4 +40,4 @@ jobs:
             --job-name nf-core-eager \
             --job-queue $AWS_JOB_QUEUE \
             --job-definition $AWS_JOB_DEFINITION \
-            --container-overrides '{"command": ["nf-core/eager", "-r '"${GITHUB_SHA}"' -profile awsfulltest --outdir s3://'"${AWS_S3_BUCKET}"'/eager/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/eager/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}'
+            --container-overrides '{"command": ["nf-core/eager", "-r '"${GITHUB_SHA}"' -profile test_full --outdir s3://'"${AWS_S3_BUCKET}"'/eager/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/eager/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}'

.github/workflows/awstest.yml

@@ -13,7 +13,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Setup Miniconda
-        uses: goanpeca/setup-miniconda@v1.0.2
+        uses: conda-incubator/setup-miniconda@v2
         with:
           auto-update-conda: true
           python-version: 3.7

.github/workflows/branch.yml

@@ -2,7 +2,7 @@ name: nf-core branch protection
 # This workflow is triggered on PRs to master branch on the repository
 # It fails when someone tries to make a PR against the nf-core `master` branch instead of `dev`
 on:
-  pull_request:
+  pull_request_target:
     branches: [master]
 
 jobs:

.github/workflows/ci.yml

@@ -26,23 +26,25 @@ jobs:
         uses: actions/checkout@v2
 
       - name: Check if Dockerfile or Conda environment changed
-        uses: technote-space/get-diff-action@v1
+        uses: technote-space/get-diff-action@v4
         with:
-          PREFIX_FILTER: |
+          FILES: |
             Dockerfile
             environment.yml
 
       - name: Build new docker image
-        if: env.GIT_DIFF
-        run: docker build --no-cache . -t nfcore/eager:2.2.1
+        if: env.MATCHED_FILES
+        run: docker build --no-cache . -t nfcore/eager:dev
 
       - name: Pull docker image
-        if: ${{ !env.GIT_DIFF }}
+        if: ${{ !env.MATCHED_FILES }}
         run: |
           docker pull nfcore/eager:dev
-          docker tag nfcore/eager:dev nfcore/eager:2.2.1
+          docker tag nfcore/eager:dev nfcore/eager:dev
 
       - name: Install Nextflow
+        env:
+          CAPSULE_LOG: none
         run: |
           wget -qO- get.nextflow.io | bash
           sudo mv nextflow /usr/local/bin/
@@ -142,22 +144,15 @@ jobs:
       - name: PMDTOOLS Test PMDtools works alone
         run: |
           nextflow run ${GITHUB_WORKSPACE} -profile test_tsv,docker --run_pmdtools
-      - name: GATK 3.5 Download resource files
-        run: |
-            mkdir -p jars/gatk_3_5
-            wget https://storage.googleapis.com/gatk-software/package-archive/gatk/GenomeAnalysisTK-3.5-0-g36282e4.tar.bz2 -P jars/gatk_3_5
-            tar xvf jars/gatk_3_5/GenomeAnalysisTK-3.5-0-g36282e4.tar.bz2 -C jars/gatk_3_5/
-            chmod +x jars/gatk_3_5/GenomeAnalysisTK.jar
-            GATK_JAR=$(readlink -f jars/gatk_3_5/GenomeAnalysisTK.jar)
       - name: GENOTYPING_UG AND MULTIVCFANALYZER Test running GATK UnifiedGenotyper and MultiVCFAnalyzer, additional VCFS
         run: |
-         nextflow run ${GITHUB_WORKSPACE} -profile test_tsv,docker --run_genotyping --gatk_ug_jar '/home/runner/work/eager/eager/jars/gatk_3_5/GenomeAnalysisTK.jar' --genotyping_tool 'ug' --gatk_out_mode 'EMIT_ALL_SITES' --gatk_ug_genotype_model 'SNP' --run_multivcfanalyzer --additional_vcf_files 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/testdata/Mammoth/vcf/JK2772_CATCAGTGAGTAGA_L008_R1_001.fastq.gz.tengrand.fq.combined.fq.mapped_rmdup.bam.unifiedgenotyper.vcf.gz' --write_allele_frequencies
+         nextflow run ${GITHUB_WORKSPACE} -profile test_tsv,docker --run_genotyping --genotyping_tool 'ug' --gatk_out_mode 'EMIT_ALL_SITES' --gatk_ug_genotype_model 'SNP' --run_multivcfanalyzer --additional_vcf_files 'https://raw.githubusercontent.com/nf-core/test-datasets/eager/testdata/Mammoth/vcf/JK2772_CATCAGTGAGTAGA_L008_R1_001.fastq.gz.tengrand.fq.combined.fq.mapped_rmdup.bam.unifiedgenotyper.vcf.gz' --write_allele_frequencies
       - name: COMPLEX LANE/LIBRARY MERGING Test running lane and library merging prior to GATK UnifiedGenotyper and running MultiVCFAnalyzer
         run: |
-         nextflow run ${GITHUB_WORKSPACE} -profile test_tsv_complex,docker --run_genotyping --gatk_ug_jar '/home/runner/work/eager/eager/jars/gatk_3_5/GenomeAnalysisTK.jar' --genotyping_tool 'ug' --gatk_out_mode 'EMIT_ALL_SITES' --gatk_ug_genotype_model 'SNP' --run_multivcfanalyzer
+         nextflow run ${GITHUB_WORKSPACE} -profile test_tsv_complex,docker --run_genotyping --genotyping_tool 'ug' --gatk_out_mode 'EMIT_ALL_SITES' --gatk_ug_genotype_model 'SNP' --run_multivcfanalyzer
       - name: GENOTYPING_UG ON TRIMMED BAM Test
         run: |
-         nextflow run ${GITHUB_WORKSPACE} -profile test_tsv,docker --run_genotyping --run_trim_bam --genotyping_source 'trimmed' --gatk_ug_jar '/home/runner/work/eager/eager/jars/gatk_3_5/GenomeAnalysisTK.jar' --genotyping_tool 'ug' --gatk_out_mode 'EMIT_ALL_SITES' --gatk_ug_genotype_model 'SNP'
+         nextflow run ${GITHUB_WORKSPACE} -profile test_tsv,docker --run_genotyping --run_trim_bam --genotyping_source 'trimmed' --genotyping_tool 'ug' --gatk_out_mode 'EMIT_ALL_SITES' --gatk_ug_genotype_model 'SNP'
       - name: BAM_INPUT Run the basic pipeline with the bam input profile, skip AdapterRemoval as no convertBam
         run: |
          nextflow run ${GITHUB_WORKSPACE} -profile test_tsv_bam,docker --skip_adapterremoval

.github/workflows/linting.yml

@@ -38,9 +38,12 @@ jobs:
         uses: actions/checkout@v2
 
       - name: Install Nextflow
+        env:
+          CAPSULE_LOG: none
         run: |
           wget -qO- get.nextflow.io | bash
           sudo mv nextflow /usr/local/bin/
+
       - uses: actions/setup-python@v1
         with:
           python-version: '3.6'
@@ -55,12 +58,19 @@ jobs:
           GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }}
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }}
-        run: nf-core -l lint_log.txt lint ${GITHUB_WORKSPACE}
+        run: nf-core -l lint_log.txt lint ${GITHUB_WORKSPACE} --markdown lint_results.md
+
+      - name: Save PR number
+        if: ${{ always() }}
+        run: echo ${{ github.event.pull_request.number }} > PR_number.txt
 
       - name: Upload linting log file artifact
         if: ${{ always() }}
         uses: actions/upload-artifact@v2
         with:
           name: linting-log-file
-          path: lint_log.txt
+          path: |
+            lint_log.txt
+            lint_results.md
+            PR_number.txt
 

.github/workflows/linting_comment.yml

@@ -0,0 +1,29 @@
+
+name: nf-core linting comment
+# This workflow is triggered after the linting action is complete
+# It posts an automated comment to the PR, even if the PR is coming from a fork
+
+on:
+  workflow_run:
+    workflows: ["nf-core linting"]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Download lint results
+        uses: dawidd6/action-download-artifact@v2
+        with:
+          workflow: linting.yml
+
+      - name: Get PR number
+        id: pr_number
+        run: echo "::set-output name=pr_number::$(cat linting-logs/PR_number.txt)"
+
+      - name: Post PR comment
+        uses: marocchino/sticky-pull-request-comment@v2
+        with:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          number: ${{ steps.pr_number.outputs.pr_number }}
+          path: linting-logs/lint_results.md
+

.github/workflows/push_dockerhub_dev.yml

@@ -0,0 +1,28 @@
+name: nf-core Docker push (dev)
+# This builds the docker image and pushes it to DockerHub
+# Runs on nf-core repo releases and push event to 'dev' branch (PR merges)
+on:
+  push:
+    branches:
+      - dev
+
+jobs:
+  push_dockerhub:
+    name: Push new Docker image to Docker Hub (dev)
+    runs-on: ubuntu-latest
+    # Only run for the nf-core repo, for releases and merged PRs
+    if: ${{ github.repository == 'nf-core/eager' }}
+    env:
+      DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+      DOCKERHUB_PASS: ${{ secrets.DOCKERHUB_PASS }}
+    steps:
+      - name: Check out pipeline code
+        uses: actions/checkout@v2
+
+      - name: Build new docker image
+        run: docker build --no-cache . -t nfcore/eager:dev
+
+      - name: Push Docker image to DockerHub (dev)
+        run: |
+          echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
+          docker push nfcore/eager:dev

.github/workflows/push_dockerhub.yml → .github/workflows/push_dockerhub_release.yml

@@ -1,16 +1,13 @@
-name: nf-core Docker push
+name: nf-core Docker push (release)
 # This builds the docker image and pushes it to DockerHub
 # Runs on nf-core repo releases and push event to 'dev' branch (PR merges)
 on:
-  push:
-    branches:
-      - dev
   release:
     types: [published]
 
 jobs:
   push_dockerhub:
-    name: Push new Docker image to Docker Hub
+    name: Push new Docker image to Docker Hub (release)
     runs-on: ubuntu-latest
     # Only run for the nf-core repo, for releases and merged PRs
     if: ${{ github.repository == 'nf-core/eager' }}
@@ -24,15 +21,7 @@ jobs:
       - name: Build new docker image
         run: docker build --no-cache . -t nfcore/eager:latest
 
-      - name: Push Docker image to DockerHub (dev)
-        if: ${{ github.event_name == 'push' }}
-        run: |
-          echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
-          docker tag nfcore/eager:latest nfcore/eager:dev
-          docker push nfcore/eager:dev
-
       - name: Push Docker image to DockerHub (release)
-        if: ${{ github.event_name == 'release' }}
         run: |
           echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
           docker push nfcore/eager:latest

.gitignore

@@ -5,8 +5,9 @@ results/
 .DS_Store
 tests/
 testing/
+testing*
 *.pyc
 main_playground.nf
 .vscode
 *.code-workspace
-nf-params.json
+nf-params.json

CHANGELOG.md

@@ -1,10 +1,26 @@
 # nf-core/eager: Changelog
 
-All notable changes to this project will be documented in this file.
-
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [2.2.2dev] - Unreleased
+
+### `Added`
+
+- Added large scale 'stress-test' profile for AWS (using de Barros Damgaard et al. 2018's 137 ancient human genomes)
+
+### `Fixed`
+
+- Fixed AWS full test profile.
+- [#587](https://github.com/nf-core/eager/issues/587) - Re-implemented AdapterRemovalFixPrefix for DeDup compatibility of including singletons
+- [#602](https://github.com/nf-core/eager/issues/602) - Added the newly avaliable GATK 3.5 conda package.
+- [#610](https://github.com/nf-core/eager/issues/610) - Create bwa_index channel when specifying circularmapper as mapper
+- Updated template to nf-core/tools 1.12
+
+### `Deprecated`
+
+- Flag `--gatk_ug_jar` has now been removed as GATK 3.5 is now avaliable within the nf-core/eager software environment.
+
 ## [2.2.1] - 2020-10-20
 
 ### `Fixed`