Skip to content
This repository was archived by the owner on Jan 27, 2020. It is now read-only.

Commit 93a0bfb

Browse files
maxulyssemaxulysse
authored
Merge pull request #698 from MaxUlysse/BTB
Add configuration profile for munin
2 parents e8469c6 + 1901ccd commit 93a0bfb

5 files changed

Lines changed: 262 additions & 52 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,15 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
1414
- [#679](https://github.com/SciLifeLab/Sarek/pull/679) - Add container for `CreateIntervalBeds`
1515
- [#692](https://github.com/SciLifeLab/Sarek/pull/692), [#697](https://github.com/SciLifeLab/Sarek/pull/697) - Add AWS iGenomes possibilities (within `conf/igenomes.conf`)
1616
- [#694](https://github.com/SciLifeLab/Sarek/pull/694) - Add monochrome and grey logos for light or dark background
17+
- [#698](https://github.com/SciLifeLab/Sarek/pull/698) - Add btb profile for munin server
1718
- [#702](https://github.com/SciLifeLab/Sarek/pull/702) - Add font-ttf-dejavu-sans-mono `2.37` and fontconfig `2.12.6` to container
1819

1920
### `Changed`
2021

2122
- [#678](https://github.com/SciLifeLab/Sarek/pull/678) - Changing VEP to v92 and adjusting CPUs for VEP
2223
- [#663](https://github.com/SciLifeLab/Sarek/pull/663) - Update `do_release.sh` script
2324
- [#671](https://github.com/SciLifeLab/Sarek/pull/671) - publishDir modes are now params
24-
- [#677](https://github.com/SciLifeLab/Sarek/pull/677) - Update docs
25+
- [#677](https://github.com/SciLifeLab/Sarek/pull/677), [#698](https://github.com/SciLifeLab/Sarek/pull/698) - Update docs
2526
- [#679](https://github.com/SciLifeLab/Sarek/pull/679) - Update old awsbatch configuration
2627
- [#682](https://github.com/SciLifeLab/Sarek/pull/682) - Specifications for memory and cpus for awsbatch
2728
- [#693](https://github.com/SciLifeLab/Sarek/pull/693) - Qualimap bamQC is now ran after mapping and after recalibration for better QC

‎conf/munin.config‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
/*
2+
* -------------------------------------------------
3+
* Nextflow config file for Sarek
4+
* -------------------------------------------------
5+
* Configuration for running on munin
6+
* -------------------------------------------------
7+
*/
8+
9+
env {
10+
NXF_WORK="/scratch"
11+
}
12+
13+
params {
14+
genome_base = params.genome == 'GRCh37' ? '/btb/references/Homo_sapiens/GATK/GRCh37/' : params.genome == 'GRCh38' ? '/btb/references/Homo_sapiens/GATK/GRCh38/' : 'References/smallGRCh37'
15+
singleCPUMem = 15.GB
16+
totalMemory = 754.GB
17+
}
18+
19+
executor.$local.cpus = 48
20+
21+
process {
22+
// Default process resources
23+
24+
// A process may use one core,
25+
cpus = 1
26+
27+
// 15 GB of memory,
28+
memory = {params.singleCPUMem}
29+
30+
// and 48 of them are allowed to be launched simultaneously.
31+
maxForks = 48
32+
33+
errorStrategy = {task.exitStatus == 143 ? 'retry' : 'terminate'}
34+
maxErrors = '-1'
35+
maxRetries = 3
36+
37+
// These processes are defined in buildReferences.nf
38+
39+
withName:BuildBWAindexes {
40+
memory = {params.totalMemory} // TODO This is likely too high
41+
}
42+
withName:BuildReferenceIndex {
43+
memory = {params.totalMemory} // TODO This is likely too high
44+
}
45+
withName:BuildSAMToolsIndex {
46+
memory = {params.totalMemory} // TODO This is likely too high
47+
}
48+
withName:BuildVCFIndex {
49+
memory = {params.totalMemory} // TODO This is likely too high
50+
}
51+
52+
// These processes are defined in main.nf
53+
54+
withName:ConcatVCF {
55+
cpus = 8
56+
}
57+
withName:CreateRecalibrationTable {
58+
cpus = 16
59+
memory = {params.totalMemory}
60+
}
61+
withName:MapReads {
62+
cpus = 48
63+
memory = {params.totalMemory}
64+
}
65+
withName:MarkDuplicates {
66+
// Actually the -Xmx value should be kept lower
67+
cpus = 16
68+
memory = {2 * params.singleCPUMem}
69+
}
70+
withName:MergeBams {
71+
cpus = 8
72+
memory = {params.totalMemory}
73+
}
74+
withName:RecalibrateBam {
75+
memory = {params.singleCPUMem * task.attempt}
76+
}
77+
withName:RunAlleleCount {
78+
memory = {params.singleCPUMem * 2 * task.attempt}
79+
}
80+
withName:RunAscat {
81+
memory = {params.singleCPUMem * 2 * task.attempt}
82+
}
83+
withName:RunBamQCmapped {
84+
cpus = 16
85+
memory = {params.totalMemory}
86+
}
87+
withName:RunBamQCrecalibrated {
88+
cpus = 16
89+
memory = {params.totalMemory}
90+
}
91+
withName:RunBcftoolsStats {
92+
}
93+
withName:RunConvertAlleleCounts {
94+
memory = {params.singleCPUMem * 2 * task.attempt}
95+
}
96+
withName:RunFastQC {
97+
cpus = 2 // FastQC is only capable of running one thread per fastq file.
98+
errorStrategy = {task.exitStatus == 143 ? 'retry' : 'ignore'}
99+
}
100+
withName:RunFreeBayes {
101+
memory = {params.singleCPUMem * task.attempt}
102+
}
103+
withName:RunHaplotypecaller {
104+
// Increase memory quadratically
105+
memory = {params.singleCPUMem * task.attempt * task.attempt}
106+
}
107+
withName:RunGenotypeGVCFs {
108+
}
109+
withName:RunManta {
110+
cpus = 48
111+
memory = {params.totalMemory}
112+
}
113+
withName:RunMultiQC {
114+
}
115+
withName:RunMutect2 {
116+
memory = {params.singleCPUMem * task.attempt}
117+
}
118+
withName:RunSamtoolsStats {
119+
}
120+
withName:RunSingleManta {
121+
cpus = 48
122+
memory = {params.totalMemory}
123+
}
124+
withName:RunSingleStrelka {
125+
cpus = 48
126+
memory = {params.totalMemory}
127+
}
128+
withName:RunSnpeff {
129+
errorStrategy = {task.exitStatus == 143 ? 'retry' : 'ignore'}
130+
memory = {params.singleCPUMem * task.attempt}
131+
}
132+
withName:RunStrelka {
133+
cpus = 48
134+
memory = {params.totalMemory}
135+
}
136+
withName:RunVEP {
137+
cpus = 48
138+
errorStrategy = {task.exitStatus == 143 ? 'retry' : 'ignore'}
139+
}
140+
}

‎docs/CONFIG.md‎

Lines changed: 73 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,22 @@ The standard ones are designed to work on a Swedish UPPMAX cluster, but can be m
1313
Every configuration file can be modified for your own use.
1414
If you want you can specify the use of a config file using `-c <config file>`
1515

16+
### [`aws-batch.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/aws-batch.config)
17+
18+
Designed for usage with AWS batch.
19+
20+
### [`base.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/base.config)
21+
22+
Define default parameters, is included into every profiles.
23+
24+
### [`binac.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/binac.config)
25+
26+
Define usage limits and Singularity for BINAC cluster in Tuebingen.
27+
28+
### [`cfc.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/cfc.config)
29+
30+
Designed for usage with Singularity on CFC at QBic.
31+
1632
### [`containers.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/containers.config)
1733

1834
Define Containers for all process.
@@ -21,15 +37,27 @@ Use in your own profile if needed.
2137

2238
### [`docker.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/docker.config)
2339

24-
Define Docker Containers for all process.
25-
Images will be pulled automatically.
26-
Use in your own profile if needed.
40+
Specify Docker options.
41+
To be used with [`containers.config`](#containersconfig)
2742

2843
### [`genomes.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/genomes.config)
2944

3045
Contain path to all references.
3146
Modify it if you want to change genome version, or the path to your references files.
3247

48+
### [`igenomes.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/igenomes.config)
49+
50+
Contain path to all AWS iGenomes references.
51+
Modify it if you want to change genome version, or the path to your references files.
52+
53+
### [`munin.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/munin.config)
54+
55+
Define usage limits and Singularity for munin server at BTB.
56+
57+
### [`resources.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/resources.config)
58+
59+
Define Generalized resource configuration for clusters.
60+
3361
### [`singularity-path.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/singularity-path.config)
3462

3563
Define path to Singularity Containers for all process.
@@ -39,14 +67,17 @@ You need to set them up before.
3967

4068
### [`singularity.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/singularity.config)
4169

42-
Define Singularity Containers for all process.
43-
Images will be pulled automatically.
44-
Use in your own profile if needed.
70+
Specify Singularity options.
71+
To be used with [`containers.config`](#containersconfig)
4572

4673
### [`travis.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/travis.config)
4774

4875
To be used for Travis (2 cpus) or on small computer for testing purpose
4976

77+
### [`uppmax-localhost.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/uppmax-localhost.config)
78+
79+
Local configuration for a UPPMAX cluster
80+
To be run on a single node
5081
### [`uppmax-slurm.config`](https://github.com/SciLifeLab/Sarek/blob/master/conf/uppmax-slurm.config)
5182

5283
Slurm configuration for a UPPMAX cluster
@@ -59,17 +90,47 @@ The default profile is `standard`, but Sarek has multiple predefined profiles wh
5990
```bash
6091
nextflow run SciLifeLab/Sarek --sample mysample.tsv -profile myprofile
6192
```
93+
awsbatch {
94+
binac {
95+
btb {
96+
cfc {
97+
docker {
98+
singularity {
99+
singularityPath {
100+
slurm {
101+
slurmDownload {
102+
standard {
103+
104+
### `awsbatch`
62105

106+
This is the profile for use with AWS Batch.
107+
108+
### `binac`
109+
110+
This is the profile for use on the german BinAC cluster.
111+
112+
### `btb`
113+
114+
This is the profile for use on the BTB server munin.
115+
116+
### `cfc`
117+
118+
This is the profile for use on the CFC cluster in Tuebingen.
63119

64120
### `docker`
65121

66122
This is the profile for docker testing on a small machine, or on Travis CI.
67123
Docker images will be pulled automatically.
68124

69-
### `standard`
125+
### `singularity`
70126

71-
This is the default profile for use on a localhost on a UPPMAX cluster with Singularity.
72-
Singularity images need to be set up.
127+
This is the profile for Singularity testing on a small machine, or on Travis CI.
128+
Singularity images will be pulled automatically.
129+
130+
### `singularityPath`
131+
132+
This is the profile for Singularity testing on a small machine.
133+
Singularity images needs to be set up.
73134

74135
### `slurm`
75136

@@ -83,10 +144,10 @@ This is another profile for use on a UPPMAX cluster using the job scheduler slur
83144
Will run the workflow on `/scratch`.
84145
Singularity images will be pulled automatically.
85146

86-
### `singularity`
147+
### `standard`
87148

88-
This is the profile for Singularity testing on a small machine, or on Travis CI.
89-
Singularity images will be pulled automatically.
149+
This is the default profile for use on a localhost on a UPPMAX cluster with Singularity.
150+
Singularity images need to be set up.
90151

91152
## Customisation
92153
The recommended way to use custom settings is to supply Sarek with an additional configuration file. You can use the files in the [`conf/`](https://github.com/SciLifeLab/Sarek/tree/master/conf) directory as an inspiration to make this new `.config` file and specify it using the `-c` flag:

‎docs/REFERENCES.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ Use `--genome smallGRCh37` to map against a small reference genome based on GRCh
101101

102102
## AWS iGenomes
103103
Sarek is using [AWS iGenomes](https://ewels.github.io/AWS-iGenomes/), which facilitate storing and sharing references.
104-
Both `GRCh37` and `GRCh38` are available with `--genome GRCh37` or `--genome GRCh38` respectively with any profile using the `conf/igenomes.config` file (eg.: `awsbatch`), or you can specify it with `-c conf/igenomes.config`, it contains all data previously detailed.
104+
Both `GRCh37` and `GRCh38` are available with `--genome GRCh37` or `--genome GRCh38` respectively with any profile using the `conf/igenomes.config` file (eg.: `awsbatch`, or `btb`), or you can specify it with `-c conf/igenomes.config`, it contains all data previously detailed.
105105

106106
## buildReferences.nf
107107

0 commit comments

Comments
 (0)
⚡