Structuring this repository

[apeltzer]

Can we maybe have a quick discussion on whether such a structure would be appropriate?:

extra/<pipeline-name>/...
reference_data/<pipeline-name>/...
testdata/<pipeline-name>/...

Extra would keep e.g. BED/PED files for certain pipelines which should be kind of unique per pipeline.
Reference_data could keep reference genome data, and can be shared in my opinion (does that make sense?)
testdata could keep per-pipeline testdata

What do others think about these points? @nf-core/admin ?

[apeltzer]

Cross linking my PR that I#d like to use as a first test point:
nf-core/exoseq#13

[maxulysse]

If I understood well, I think @ewels was thinking about something like:

extra/
reference/
testdata/

But with a different branch for each pipeline

[apeltzer]

One particular problem I see is that the repository might become too big for hosting on GitHub.
Do you think that could be a problem?
https://help.github.com/articles/what-is-my-disk-quota/
Branches sound great - I suppose that should keep things clean

[maxulysse]

I suppose that if files are getting too big, we could try git-LFS

[maxulysse]

And we should list the different branches in the README.md on master

[ewels]

The branches idea came from @pditommaso - that way we can have a git submodule for a single test dataset without having to download the test data for every nf-core pipeline. Each branch should be an orphan - master can just have a readme with instructions.

Total repository size could become an issue, yes. I'm not sure how LFS really helps - GitHub still has the 1GB limit, even with LFS..? I guess we just need to keep things slim!

[ewels]

Other stuff - would be nice to have in the branch readme a list of all files and their sizes perhaps? Maybe even an md5sum file? Maybe this is overkill..

[maxulysse]

I think it might be overkill for test data.
But we can probably set up a script to pull all branches and generate the README, even with the md5sum ;-)

[tiagochst]

Git LFS would help to add files over 100MB, without git LFS, the commit will be blocked for files over 100MB. I believe there is a warning for files between 50MB and 100MB saying the max is 50MB.

[maxulysse]

By the way, we set this up for Sarek:
https://github.com/SciLifeLab/Sarek-data

I think you'll all see where the inspiration comes from ;-)

[apeltzer]

Regarding git lfs: yes it would help but we should try to keep things as small as possible in all cases anyways. The Sarek way looks fine to me and the ExoSeq pipeline will do the same in the next days once I’m back at work :)

[anfederico]

Looking forward to this! Would be an amazing resource. I have a good set I've been using for scRNA-seq pipelines.

[ewels]

I think we can close this issue now - we've started building different branches for the different pipelines and it seems to be working well!