Checking for file existence in S3 broken since league/flysystem-aws-s3-v3 3.12.1

[ptkoz]

Bug Report

Q	A
Flysystem Version	3.12.2
Adapter Name	Aws S3 (v3) Adapter
Adapter version	3.12.1 & 3.12.2

Summary

Following code will:

• return false for version up to 3.10.3
• throw an exception for 3.12.1 & 3.12.2

$flysytsem = new \League\Flysystem\Filesystem(
    new \League\Flysystem\AwsS3V3\AwsS3V3Adapter(
        new \Aws\S3\S3Client([
            'region' => 'PUT AWS REGION HERE',
            'version' => 'latest',
            'credentials' => [
                'key' => 'PUT AWS KEY ID HERE',
                'secret' => 'PUT AWS SECRET HERE',
            ]
        ]),
        'PUT S3 BUCKET NAME HERE'
    ),
);

var_dump($flysytsem->fileExists('/some/non-existing/file'));

Please note that for existing files, newest versions also correctly return true, without an exception. I've tested this with various aws-sdk versions and it seems to be unrelated (tested various versions from 3.236.6 to 3.257.5, same behaviour).

Also the calling identity has full access to the S3 bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::bucket-name/*"
        }
    ]
}

How to reproduce

1. Clone bare-minimum example repository here: https://github.com/ptkoz/flysystem-aws-s3-v3-file-exists-bug
2. Run composer install
3. Run php test.php
4. Observe the issue

Then:
5. Downgrade version number in composer.json to ~3.10.0
6. Run composer update
7. Run php test.php
8. Observe it works

The exception

Posting full exception trace seems unnecessary. There are various Flysystem / AWS / Guzzle exceptions in the stack. The issue though seems to originate in 403 response from AWS api for HEAD request, as follows:

Uncaught GuzzleHttp\Exception\ClientException: Client error: `HEAD https://xxxxx.s3.eu-west-1.amazonaws.com/some/non-existing/file` resulted in a `403 Forbidden` response in /flysystem-aws-s3-v3-file-exists-bug/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113

[ptkoz]

git bisect shows the issue has been introduced in this commit: thephpleague/flysystem-aws-s3-v3@ea10034

[ptkoz]

Raading aws-sdk docs and looking at their code, I think the issue originates there. The major difference between doesObjectExistV2 and doesObjectExist is that the latter does this:

catch (S3Exception $e) {
    if ($e->getAwsErrorCode() == 'AccessDenied') {
        return true;
    }

    if ($e->getStatusCode() >= 500) {
        throw $e;
    }

    return false;
}

effectively interpreting any response code < 500 as "object does not exist", while the former explicitly requires 404:

catch (S3Exception $e) {
    if ($includeDeleteMarkers
        && $this->useDeleteMarkers($e)
    ) {
        return true;
    }
            
    if ($e->getStatusCode() === 404) {
        return false;
    }

    throw $e;
}

I have checked that both make the very same request and receive very same 403 response. I guess I'll raise an issue for the aws-sdk, but still I think flysystem should fall back to using objectDoesExists until V2 is fixed in upstream, shouldn't it?

[frankdejonge]

Well, that's tricky. You've raised this behaviour as an issue, while others have raised an issue for the previous behaviour, leaving the project in a split. They got their PR merged with AWS (see: aws/aws-sdk-php#2424), so I think it's better to fix the behaviour upstream. For now, I'd recommend extending the class to put the old version in the extension, or to pin your dependency until the upstream is fixed.

[ptkoz]

After reviewing AWS api documentations, this issue in aws-sdk and this issue I have come to conclusion that something must be wrong with my permissions. Closing the bug, as it turned out to be true, to get ListBucket permissions, specifying "arn:aws:s3:::bucket-name/*" as resource is not enough, one needs to explicitly declare:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::bucket_name",
                "arn:aws:s3:::bucket_name/*"
            ]
        }
    ]
}

Once ListBucket permission was correctly granted, the exception is gone.