Support for AWS IAM role

[fstrozzi]

Hello,
it would be nice if Nextflow could support running EC2 instances with a specified IAM instance profile, so that new instances will be automatically assigned the correct permissions to e.g. read / write on S3 buckets, without the need to explicitly manage AWS access keys which is not advised for security reasons.

Thanks!

[pditommaso]

I don't know this feature. Can you provide an AWS doc reference about it?

[fstrozzi]

Sure, here is the link: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

Once a role granting the necessary permissions has been defined in IAM, an Instance Profile for that role can be created and associated to EC2 instances when they are launched.

[pditommaso]

Regarding this, it should be enough to verify that a IAM Role has been provided checking the instance metadata

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<role name>

[fstrozzi]

Once an EC2 instance is launched with an IAM role associated, every call to AWS services (via the aws cli or the AWS JDK) will just work according to the role permissions. Getting the associated role will be useful for logging purposes for example, but the point of this issue was more when an elastic cluster is created by NF, I think there should be an option to specify the IAM profile to use to launch EC2 instances.

[pditommaso]

Very likely the AWS SDK automatically fetch the accessId and the secretKey from that URL. The relevant changes in the NF code is related to the AWS client creation which currently is expecting the AWS credential.

I agree to log the current IAM Role, however the NF approach is to not require config options if they are not needed !

I would propose to check the above metadata URL to determine if NF is running in an Ec2 instance with a configured IAM role, if yes it should read the role name and trace in the log information.

[pditommaso]

This patch allows access/secret keys are no mandatory any more and NF is able to use the IAM instance profile define in the hosting Ec2 instance.

Moreover it's possible to specify the setting iamProfile in the cloud configuration setting to provide the IAM profile role name to use when deploying a AWS cluster.

I'm only a bit concerned about the naming, is iamProfile consistent? AWS documentation/tools are confusing. In the command line and the API it is name IAM instance profile, in the EC2 dashboard is referred IAM Role, in the instance metadata IAM security credentials.

[fstrozzi]

Yes, it's a bit confusing. That's because in IAM you first create a Role, then to associate it to an Instance you need to create an Instance Profile. I think IAM instance profile should be pretty stable though.