Skip to content

Umi test #614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
FriederikeHanssen merged 31 commits into from
Jul 17, 2022
Merged

Umi test #614

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
583a91e
Started to work on the umi stuff
WackerO Jun 28, 2022
e7da199
Fixed bug in modules.config
WackerO Jun 30, 2022
a9889a1
Code seems to run now, but had to comment ou the meta id tag from sam…
WackerO Jul 1, 2022
48502f9
Reenabled meta.id, fixed DRAGMAP args
WackerO Jul 1, 2022
2563e1b
Bugfixing
WackerO Jul 1, 2022
037e063
Finally, it works
WackerO Jul 4, 2022
708a7be
Merge branch 'dev' of https://github.com/nf-core/sarek into umi
WackerO Jul 4, 2022
2d59ccb
Removed changes to module
WackerO Jul 4, 2022
6c02d86
Updated test_umi.yaml
WackerO Jul 4, 2022
99da99d
This newer container version should make fgbio work
WackerO Jul 11, 2022
be85845
Merge branch 'dev' of https://github.com/nf-core/sarek into umi
WackerO Jul 11, 2022
2fffc69
Restored fgbio module, updated test_umi.yml
WackerO Jul 11, 2022
406a919
Updated fgbio
WackerO Jul 12, 2022
bec23a4
Merge branch 'dev' of https://github.com/nf-core/sarek into umi
WackerO Jul 12, 2022
65a625b
Bug-fixing
WackerO Jul 12, 2022
e289b6e
Still bugfixing
WackerO Jul 13, 2022
26bdb63
Commented out the umi ci test as it currently produces an error in py…
WackerO Jul 13, 2022
f7a5aea
Merge branch 'dev' of https://github.com/nf-core/sarek into umi
WackerO Jul 13, 2022
3d657c5
Reverted module changes
WackerO Jul 13, 2022
4c1c710
Updated the fgbio modules
WackerO Jul 14, 2022
d90ce80
Merge branch 'dev' of https://github.com/nf-core/sarek into umi
WackerO Jul 14, 2022
3cc7e7a
Testing if the new containers make umi work
WackerO Jul 14, 2022
6146e33
Nope, still snappy error
WackerO Jul 14, 2022
3ca687a
Merge branch 'dev' of https://github.com/nf-core/sarek into umi
WackerO Jul 15, 2022
a5e05f3
Updated modules and reenabled umi in ci.yml; hopefully the test passe…
WackerO Jul 15, 2022
04ddfd4
Updated test checksums for umi
WackerO Jul 15, 2022
5587747
Fixed typo in md5sum
WackerO Jul 15, 2022
41bfe6f
Merge remote-tracking branch 'upstream/dev' into umi
FriederikeHanssen Jul 17, 2022
e053000
update umi checksums
FriederikeHanssen Jul 17, 2022
0e40b6a
no need to overwrite the dragmap prefix, it is the same
FriederikeHanssen Jul 17, 2022
eb07bf0
remove checksum for collated bam file
FriederikeHanssen Jul 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ jobs:
- "tiddit"
- "trimming"
- "tumor_normal_pair"
- "umi"
- "variantcalling_channel"

steps:
Expand Down
15 changes: 10 additions & 5 deletions conf/modules.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,7 @@ process {

withName: 'BAM2FASTQ' {
ext.args = '-T RX'
ext.when = { umi_read_structure }
ext.when = { params.umi_read_structure }
}

withName: 'CALLUMICONSENSUS' {
Expand All @@ -238,6 +238,12 @@ process {
ext.args2 = '-bS'
ext.prefix = {"${meta.id}.umi_unsorted"}
}

withName: ".*DRAGMAP_ALIGN" {
ext.args = { "--RGSM ${meta.read_group}" }
ext.args2 = '-bS'
ext.prefix = {"${meta.id}.umi_unsorted"}
Comment thread
FriederikeHanssen marked this conversation as resolved.
}
}

withName: 'SAMBLASTER' {
Expand All @@ -247,17 +253,16 @@ process {

// MAPPING
if (params.step == 'mapping') {
withName: ".*GATK4_MAPPING:BWAMEM1_MEM" {
withName: ".*BWAMEM1_MEM" {
ext.when = { params.aligner == "bwa-mem" }
}
withName: ".*GATK4_MAPPING:BWAMEM2_MEM" {
withName: ".*BWAMEM2_MEM" {
ext.when = { params.aligner == "bwa-mem2" }
}

withName: ".*GATK4_MAPPING:DRAGMAP_ALIGN" {
withName: ".*DRAGMAP_ALIGN" {
ext.when = { params.aligner == "dragmap" }
ext.args = { "--RGSM ${meta.read_group}" }
Comment thread
WackerO marked this conversation as resolved.

}

withName: "NFCORE_SAREK:SAREK:GATK4_MAPPING:(BWAMEM.*_MEM|DRAGMAP_ALIGN)" {
Expand Down
4 changes: 2 additions & 2 deletions modules/nf-core/modules/fgbio/fastqtobam/main.nf
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

181 changes: 91 additions & 90 deletions tests/test_umi.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,93 +1,94 @@
- name: Run UMI Quiaseq test
command: nextflow run main.nf -profile test,umi_quiaseq,docker
- name: Run UMI test
command: nextflow run main.nf -profile test,umi,docker
tags:
- umi_quiaseq
- umi
files:
- path: results/fastqtobam/1234N-SRR7545951_umi_converted.bam
md5sum: 1473c98be04aff10f80e2946b111c459
- path: results/bam2fastq/1234N-SRR7545951_interleaved.fq.gz
md5sum: 17c0cb479e8117016e9f0d835d4df50f
- path: results/callumiconsensus/1234N-SRR7545951_umi-consensus.bam
md5sum: 3be1adf27d473de12cab895423a5f3ed
- path: results/cat/1234N-SRR7545951_1.merged.fastq.gz
md5sum: c1ed636a11b3031c72f3d4c84a7a0949
- path: results/cat/1234N-SRR7545951_2.merged.fastq.gz
md5sum: 0bbef631b8de80f2abe441de713cbb49
- path: results/groupreadsbyumi/1234N-SRR7545951_umi-grouped.bam
md5sum: e3bb348056513c46d7a60df4d7a8cc97
- path: results/groupreadsbyumi/1234N-SRR7545951_umi_histogram.txt
md5sum: 2e1e9a774135675a3d1a3b0a88d29c53
- path: results/samblaster/1234N-SRR7545951_unsorted_tagged.bam
md5sum: daa51939768c256315ce700dabaccad6
- path: results/samtools/1234N-SRR7545951.map_map.bam
md5sum: 741de784bb83815fab8b2d23d88c74d7
- path: results/samtools/1234N-SRR7545951.map_unmap.bam
md5sum: 31627b2d163da372584bb78258c046e5
- path: results/samtools/1234N-SRR7545951.mapped_1.fq.gz
md5sum: 709872fc2910431b1e8b7074bfe38c67
- path: results/samtools/1234N-SRR7545951.mapped_2.fq.gz
md5sum: 709872fc2910431b1e8b7074bfe38c67
- path: results/samtools/1234N-SRR7545951.mapped_other.fq.gz
md5sum: 709872fc2910431b1e8b7074bfe38c67
- path: results/samtools/1234N-SRR7545951.mapped_singleton.fq.gz
md5sum: 709872fc2910431b1e8b7074bfe38c67
- path: results/samtools/1234N-SRR7545951.unmap_map.bam
md5sum: 42fb9dbb5a157c194c77b1de5b4f3b39
- path: results/samtools/1234N-SRR7545951.unmap_unmap.bam
md5sum: 2075e48cd007f29c8f6fdc518e587e27
- path: results/samtools/1234N-SRR7545951.unmapped_1.fq.gz
md5sum: 1f95ae5bbc7345372005b6673884f853
- path: results/samtools/1234N-SRR7545951.unmapped_2.fq.gz
md5sum: 9902e27a99c2a628d184cb91759d6ff7
- path: results/samtools/1234N-SRR7545951.unmapped_other.fq.gz
md5sum: 709872fc2910431b1e8b7074bfe38c67
- path: results/samtools/1234N-SRR7545951.unmapped_singleton.fq.gz
md5sum: 709872fc2910431b1e8b7074bfe38c67

- name: Run UMI TSO test
command: nextflow run main.nf -profile test,umi_tso,docker
tags:
- umi_tso
- umi
files:
- path: results/fastqtobam/1234N-HT1080_umi_converted.bam
md5sum: aaf4b9a80f4b780d93820df8b536c531
- path: results/bam2fastq/1234N-HT1080_interleaved.fq.gz
md5sum: b037fe8e180e805226485d9bea4a69e5
- path: results/callumiconsensus/1234N-HT1080_umi-consensus.bam
md5sum: 807bfe6a82eabdf29aaf8bc6a832f0cd
- path: results/cat/1234N-HT1080_1.merged.fastq.gz
md5sum: c8f0d1308df4c46c0716ccfe76f78fb4
- path: results/cat/1234N-HT1080_2.merged.fastq.gz
md5sum: 1b42767b2d5dbc99e447c41d6d2418ab
- path: results/groupreadsbyumi/1234N-HT1080_umi-grouped.bam
md5sum: c9c76b410ba208fea76c79cecfb45241
- path: results/groupreadsbyumi/1234N-HT1080_umi_histogram.txt
md5sum: 3dec92ae49763c10c0b2e79827d25ffb
- path: results/samblaster/1234N-HT1080_unsorted_tagged.bam
md5sum: 628bd970fa00615fb9f841bad7c8f597
- path: results/samtools/1234N-HT1080.map_map.bam
md5sum: 2689e640c3ededf70228a44ee4b1de48
- path: results/samtools/1234N-HT1080.map_unmap.bam
md5sum: 37300fea69f7c400220a03ca5854d04b
- path: results/samtools/1234N-HT1080.mapped_1.fq.gz
md5sum: 709872fc2910431b1e8b7074bfe38c67
- path: results/samtools/1234N-HT1080.mapped_2.fq.gz
md5sum: 709872fc2910431b1e8b7074bfe38c67
- path: results/samtools/1234N-HT1080.mapped_other.fq.gz
md5sum: 709872fc2910431b1e8b7074bfe38c67
- path: results/samtools/1234N-HT1080.mapped_singleton.fq.gz
md5sum: 709872fc2910431b1e8b7074bfe38c67
- path: results/samtools/1234N-HT1080.unmap_map.bam
md5sum: 6267d0ed8274e1fa7aad8d1e71d98696
- path: results/samtools/1234N-HT1080.unmap_unmap.bam
md5sum: 32ed07aa23900f5722e5983a646a5ac2
- path: results/samtools/1234N-HT1080.unmapped_1.fq.gz
md5sum: d9a7a279c05563a37d9a5a6938be68fb
- path: results/samtools/1234N-HT1080.unmapped_2.fq.gz
md5sum: afe191dcd48360f96bdb583fa60ff400
- path: results/samtools/1234N-HT1080.unmapped_other.fq.gz
md5sum: 709872fc2910431b1e8b7074bfe38c67
- path: results/samtools/1234N-HT1080.unmapped_singleton.fq.gz
md5sum: 709872fc2910431b1e8b7074bfe38c67
- path: results/fastqtobam/test-test_L1_umi_converted.bam
md5sum: ce67e3b1f46302127cd07cb3c2d2e739
- path: results/bam2fastq/test-test_L1_interleaved.fq.gz
md5sum: 7a57a8ab53dd1d799cca67a85c47ccd9
- path: results/callumiconsensus/test-test_L1_umi-consensus.bam
md5sum: 8d52eadfa1919eb01c795c29b3f3a2f3
- path: results/cat/test-test_L1_1.merged.fastq.gz
md5sum: 56689dad7749b989c003a18f83445cae
- path: results/cat/test-test_L1_2.merged.fastq.gz
md5sum: bede34e291fd70f6e5fb5ad576617272
- path: results/groupreadsbyumi/test-test_L1_umi-grouped.bam
md5sum: 14b02a0279f715586d3671d184a83d6b
- path: results/groupreadsbyumi/test-test_L1_umi_histogram.txt
md5sum: d17fd167b2a765d46e4b01bf08ece01b
- path: results/samblaster/test-test_L1_unsorted_tagged
md5sum: ba3425e6722d6f6cc36e6dec42b19207
- path: results/samtools/test-test_L1.map_map.bam
md5sum: f29dc564311e592941bdf152dd722b4f
- path: results/samtools/test-test_L1.map_unmap.bam
md5sum: 6518cea4d9c0bdb17af66932b68e0ad9
- path: results/samtools/test-test_L1.bam
md5sum: 41f807e2f3be27a6054a1b7ab4a438b2
# TODO: The following files are not produced anymore
# - path: results/samtools/test-test_L1.mapped_1.fq.gz
# md5sum: 709872fc2910431b1e8b7074bfe38c67
# - path: results/samtools/test-test_L1.mapped_2.fq.gz
# md5sum: 709872fc2910431b1e8b7074bfe38c67
# - path: results/samtools/test-test_L1.mapped_other.fq.gz
# md5sum: 709872fc2910431b1e8b7074bfe38c67
# - path: results/samtools/test-test_L1.mapped_singleton.fq.gz
# md5sum: 709872fc2910431b1e8b7074bfe38c67
# - path: results/samtools/test-test_L1.unmap_map.bam
# md5sum: 9479aabdbbfdc8f5c79f8daa1011e9fb
# - path: results/samtools/test-test_L1.unmap_unmap.bam
# md5sum: 68bcb7f856c1933b81c4733da2fcd8df
# - path: results/samtools/test-test_L1.unmapped_1.fq.gz
# md5sum: 1f95ae5bbc7345372005b6673884f853
# - path: results/samtools/test-test_L1.unmapped_2.fq.gz
# md5sum: 9902e27a99c2a628d184cb91759d6ff7
# - path: results/samtools/test-test_L1.unmapped_other.fq.gz
# md5sum: 709872fc2910431b1e8b7074bfe38c67
# - path: results/samtools/test-test_L1.unmapped_singleton.fq.gz
# md5sum: 709872fc2910431b1e8b7074bfe38c67
# - name: Run UMI TSO test
# command: nextflow run main.nf -profile test,umi_tso,docker
# tags:
# - umi_tso
# - umi
# files:
# - path: results/fastqtobam/1234N-HT1080_umi_converted.bam
# md5sum: aaf4b9a80f4b780d93820df8b536c531
# - path: results/bam2fastq/1234N-HT1080_interleaved.fq.gz
# md5sum: b037fe8e180e805226485d9bea4a69e5
# - path: results/callumiconsensus/1234N-HT1080_umi-consensus.bam
# md5sum: 807bfe6a82eabdf29aaf8bc6a832f0cd
# - path: results/cat/1234N-HT1080_1.merged.fastq.gz
# md5sum: c8f0d1308df4c46c0716ccfe76f78fb4
# - path: results/cat/1234N-HT1080_2.merged.fastq.gz
# md5sum: 1b42767b2d5dbc99e447c41d6d2418ab
# - path: results/groupreadsbyumi/1234N-HT1080_umi-grouped.bam
# md5sum: c9c76b410ba208fea76c79cecfb45241
# - path: results/groupreadsbyumi/1234N-HT1080_umi_histogram.txt
# md5sum: 3dec92ae49763c10c0b2e79827d25ffb
# - path: results/samblaster/1234N-HT1080_unsorted_tagged.bam
# md5sum: 628bd970fa00615fb9f841bad7c8f597
# - path: results/samtools/1234N-HT1080.map_map.bam
# md5sum: 2689e640c3ededf70228a44ee4b1de48
# - path: results/samtools/1234N-HT1080.map_unmap.bam
# md5sum: 37300fea69f7c400220a03ca5854d04b
# - path: results/samtools/1234N-HT1080.mapped_1.fq.gz
# md5sum: 709872fc2910431b1e8b7074bfe38c67
# - path: results/samtools/1234N-HT1080.mapped_2.fq.gz
# md5sum: 709872fc2910431b1e8b7074bfe38c67
# - path: results/samtools/1234N-HT1080.mapped_other.fq.gz
# md5sum: 709872fc2910431b1e8b7074bfe38c67
# - path: results/samtools/1234N-HT1080.mapped_singleton.fq.gz
# md5sum: 709872fc2910431b1e8b7074bfe38c67
# - path: results/samtools/1234N-HT1080.unmap_map.bam
# md5sum: 6267d0ed8274e1fa7aad8d1e71d98696
# - path: results/samtools/1234N-HT1080.unmap_unmap.bam
# md5sum: 32ed07aa23900f5722e5983a646a5ac2
# - path: results/samtools/1234N-HT1080.unmapped_1.fq.gz
# md5sum: d9a7a279c05563a37d9a5a6938be68fb
# - path: results/samtools/1234N-HT1080.unmapped_2.fq.gz
# md5sum: afe191dcd48360f96bdb583fa60ff400
# - path: results/samtools/1234N-HT1080.unmapped_other.fq.gz
# md5sum: 709872fc2910431b1e8b7074bfe38c67
# - path: results/samtools/1234N-HT1080.unmapped_singleton.fq.gz
# md5sum: 709872fc2910431b1e8b7074bfe38c67
4 changes: 3 additions & 1 deletion workflows/sarek.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,8 +386,10 @@ workflow SAREK {
umi_read_structure,
params.group_by_umi_strategy)

bamtofastq = CREATE_UMI_CONSENSUS.out.consensusbam.map{meta, bam -> [meta,bam,[]]}

// convert back to fastq for further preprocessing
ALIGNMENT_TO_FASTQ_UMI(CREATE_UMI_CONSENSUS.out.consensusbam, [])
ALIGNMENT_TO_FASTQ_UMI(bamtofastq, [])

ch_reads_fastp = ALIGNMENT_TO_FASTQ_UMI.out.reads

Expand Down
⚡