cannot stage in files from s3 compatible storage service with non-aws endpoint with newest Nextflow/nf-amazon version

[katgorski]

Bug report

Expected behavior and actual behavior

I am trying to stage in files stored on Wasabi (https://wasabi.com/), which is s3 compatible. The code below fails with Nextflow version 23.10.1, but runs successfully with the command NXF_VER=23.04.5 nextflow run main.nf -c nextflow.config (which uses nf-amazon@1.16.2).

Steps to reproduce the problem

main.nf

workflow {
    WASABI_STAGE_WASABI_PUB(params.wasabi_path)
}

process WASABI_STAGE_WASABI_PUB {

    publishDir "${params.wasabi_publish_dir}", saveAs: { filename -> "${new Date().format('yyyyMMddHHmmss')}_${task.executor}_${filename}" }

    input:
    path(wasabi_path)

    output:
    path("wasabi_to_wasabi.txt")

    script:
    """
    ls -al . > wasabi_to_wasabi.txt
    """

}

nextflow.config

params {
    wasabi_path = 's3://s3-bucket/s3-filepath/s3-file'
    wasabi_publish_dir = 's3://s3-bucket/nextflow'
}

aws {
    accessKey = '---'
    secretKey = '---'
    client {
        endpoint = 'https://s3.us-east-2.wasabisys.com'
        s3PathStyleAccess = true
    }
}

Command: nextflow run main.nf -c nextflow.config

Program output

N E X T F L O W  ~  version 23.10.1
Launching `main.nf` [silly_poitras] DSL2 - revision: 4556fd0279
[-        ] process > WASABI_STAGE_WASABI_PUB -
WARN: Unable to get file attributes file: s3://s3-bucket/s3-filepath/s3-file -- Cause: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exi
[-        ] process > WASABI_STAGE_WASABI_PUB -
WARN: Unable to get file attributes file: s3://s3-bucket/s3-filepath/s3-file -- Cause: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: JC8X87T8A3BT3Y8E; S3 Extended Request ID: sajJzdE1PhC2JTBwH3W7Qc4g8GfwKjPIlgGv73eeHe8IS2m7zf+m5eiGJxRa8CpYTZU2l61t61A=; Proxy
[-        ] process > WASABI_STAGE_WASABI_PUB -
WARN: Unable to get file attributes file: s3://s3-bucket/s3-filepath/s3-file -- Cause: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: JC8X87T8A3BT3Y8E; S3 Extended Request ID: sajJzdE1PhC2JTBwH3W7Qc4g8GfwKjPIlgGv73eeHe8IS2m7zf+m5eiGJxRa8CpYTZU2l61t61A=; Proxy: null), S3 Extended Request ID: sajJzdE1PhC2JTBwH3W7Qc4g8GfwKjPIlgGv73eeHe8IS2m7zf+m5eiGJxRa8CpYTZU2l61t61A=
ERROR ~ Error executing process > 'WASABI_STAGE_WASABI_PUB'

Caused by:
  The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: JC8H0E7X9W2WAEM0; S3 Extended Request ID: pNZ62Q25W+xIKP5O4iyld4b3yu9EMd64bjRGo3qJRoRlrqmlgqljuIQQtXlvI8xI268LJkQAmY4=; Proxy: null)


 -- Check '.nextflow.log' file for details

nextflow log:
nextflow.log.23.10.1.txt

Environment

• Nextflow version: 23.10.1 (nf-amazon@2.1.4)
• Java version: 17.0.10
• Operating system: macOS Sonoma 14.4.1 (however, this also occurs on Ubuntu 22.04.4 LTS)
• Bash version: zsh 5.9 (x86_64-apple-darwin23.0) (also occurs on GNU bash 5.1.16)

Additional context

Again, this works as expected with NXF_VER=23.04.5 nextflow run main.nf -c nextflow.config (Java, OS, zsh version the same). Logs for that execution:
nextflow.log.23.04.5.txt

[bentsherman]

This error usually means some part of your credentials are missing. Do you also have a session token in addition to access key and secret key?

[katgorski]

This error usually means some part of your credentials are missing. Do you also have a session token in addition to access key and secret key?

I do not. I drove myself crazy trying to determine if the credentials were OK or not, but passing in just the keys in 23.04 was fine (same credentials across the 23.04 and 23.10 executions). I know nf-amazon was overhauled between the two releases, but I'm not familiar with the interfaces used for interacting with aws. Theoretically this could be something specific to my endpoint, but I'm not sure.

[katgorski]

Hi, reviving this a bit, just to check in--it seems like this could be related to #4811 (which has a much more significant trace and more conversation, thanks to @rjb32 in that thread) and that it may be resolved with an upgrade to Amazon Java SDK v2 (#4741). Is there any status update on any of the three issues?

[Yifei-Wan]

I am experiencing the exact same issue. I tested the endpoint using the AWS S3 CLI directly, and it works.

• Config

aws {
    accessKey = 'xxxxxx'
    secretKey = 'xxxxxxx'
    client {
        endpoint = 'https://s3.wasabisys.com'
        s3PathStyleAccess = true 
        protocol = 'https'
    }
}

• Log

Jul-30 17:00:43.900 [Actor Thread 4] DEBUG nextflow.cloud.aws.config.AwsConfig - AWS S3 config properties: {endpoint=https://s3.wasabisys.com, s_3_path_style_access=true, protocol=https, max_error_retry=5}

• Error

Caused by:
  The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: MACB6GK488WJD8BV; S3 Extended Request ID: Auut3p4QkWv6jJgYiveX7sRLQNtBoDdE5s7qARDUyJJ9UShYXmvrMxd55tdfZm9yUFQYkGycv4RxyCuzK3DECQ==; Proxy: null)

[bentsherman]

Possibly fixed by #5779

[katgorski]

Possibly fixed by #5779

have that on watch, plan on looking at the new edge release asap 🤞

[katgorski]

@bentsherman can confirm that this is no longer an issue in the new edge release!

[bentsherman]

Awesome 😄