How to share results and store them

[FriederikeHanssen]

Currently, all results are stored indefinitely in the S3 output bucket. We should discuss how to handle them, whether they should be deleted and when, and how to share them.

[ewels]

If possible, I would like to store at least one set of results for each pipeline and embed / link to these from the nf-core pipeline page. If we can afford it, it would be nice to keep a set of results for each release of each pipeline.

[ewels]

This could be good for the website: https://github.com/minio/minio-js

[FriederikeHanssen]

There is https://aws.amazon.com/glacier/ which seems to be cheaper alternative to the normal s3 buckets at costs for slower access. We discussed at some point whether it is feasible to use this for longer time storage, but never followed up on it. Might be an alternativ?

[ewels]

That won't work sadly, as if you want to view any files in Glacier you have to request and then wait several hours / days. So it doesn't work with people wanting to browse and view the files.

We should be able to set up automatic migration to S3 IA though (infrequent access). I think that should still work in the way that we want and might reduce costs for stuff that no-one looks at, like files from old releases.

[FriederikeHanssen]

Here is a cost overview: https://aws.amazon.com/s3/pricing/ . S3 IA: $0.0125 per GB vs S3 standard: $0.023 per GB.

[ewels]

I think let's keep it simple for now and just stick with regular S3. Down the road if we still have credits and still have this running we can always revisit this.

[jamesoff]

I think let's keep it simple for now and just stick with regular S3. Down the road if we still have credits and still have this running we can always revisit this.

If it's likely that the data will be "cold" after some time, you can use Lifecycle Policies to put it into S3-IA. It may also be worth looking at S3 Intelligent Tiering, which will move data between Standard and IA according to how it's used. There's a charge per object which it manages, but otherwise the cost is just the storage of the objects in whatever tier they're in.