S3 client believes every object exists.

[manarth]

Describe the bug

When invoking 'doesObjectExist()` with the S3 client, the API returns TRUE for non-existing objects.

Expected Behavior

The API should return TRUE if the specified bucket and key exists.
The API should return FALSE if the specified bucket does not exist, or if the specified bucket and key do not exist.

Current Behavior

The API always returns true.

Reproduction Steps

$params = [
  'Bucket' => 'my-bucket-which-exists',
  // Non-existent key name.
  'Key' => 'sdsgfdsgregerdfdgdd',
];
$result = $s3Client->doesObjectExist($params['Bucket'], $params['Key']);

Possible Solution

The method checkExistenceWithCommand() in the S3ClientTrait executes a defined command.
If the command does not throw an exception, TRUE is returned.

    /**
     * Determines whether or not a resource exists using a command
     *
     * @param CommandInterface $command Command used to poll for the resource
     *
     * @return bool
     * @throws S3Exception|\Exception if there is an unhandled exception
     */
    private function checkExistenceWithCommand(CommandInterface $command)
    {
        try {
            $result = $this->execute($command);
            return true;
        } catch (S3Exception $e) {
            if ($e->getAwsErrorCode() == 'AccessDenied') {
                return true;
            }
            if ($e->getStatusCode() >= 500) {
                throw $e;
            }
            return false;
        }
    }

The doesObjectExist() method uses a HeadObject command to check if the bucket exists.

    /**
     * @see S3ClientInterface::doesObjectExist()
     */
    public function doesObjectExist($bucket, $key, array $options = [])
    {
        return $this->checkExistenceWithCommand(
            $this->getCommand('HeadObject', [
                    'Bucket' => $bucket,
                    'Key'    => $key
                ] + $options)
        );
    }

When executing a HeadObject command for a non-existing key, the command does not throw an Exception.
Instead, it returns the 404 response as an Aws\Result object.

class Aws\Result#2316 (2) {
  private $data =>
  array(1) {
    '@metadata' =>
    array(4) {
      'statusCode' =>
      int(404)
      'effectiveUri' =>
      string(55) "https://<redacted>.s3.amazonaws.com/sdsgfdsgregerdfdgdd"
      'headers' =>
      array(5) {
        ...
      }
      'transferStats' =>
      array(1) {
        ...
      }
    }
  }
  private $monitoringEvents =>
  array(0) {
  }
}

Additional Information/Context

The most significant end-user impact is that using mkdir() with the S3 StreamWrapper always fails, because it always believes the (pseudo)-directory exists.

Solution options may include:

• Within checkExistenceWithCommand(), test for a 404 result (in addition to detecting an exception).
• Within the HeadObject command, throw an Exception for non-existing objects.

SDK version used

3.222.6

Environment details (Version of PHP (php -v)? OS name and version, etc.)

PHP 7.4.29 (cli), on Ubuntu Bionic 18.04.

[stobrien89]

Hi @manarth,

Thanks for raising this. We recently implemented doesObjectExistV2 and doesBucketExistV2, which address some of the faulty assumptions made by the first implementations. I did take into consideration the customizations that rely on the original implementations, but realized changing them could be a breaking change for customers and did not know the extent to which customers were being impacted.

I think the most sensible thing to do at this point would be to add an optional argument for usage of the newer existence methods. I'll shoot to have a PR up for this within a week.

[manarth]

Hi @stobrien89 – the unexpected behaviour also currently exists for the doesObjectExistV2 method (I haven't been able to trace it to a specific change yet though).

[manarth]

My current local patch checks for a 404 response:

    /**
     * Determines whether or not a resource exists using a command
     *
     * @param CommandInterface $command Command used to poll for the resource
     *
     * @return bool
     * @throws S3Exception|\Exception if there is an unhandled exception
     */
    private function checkExistenceWithCommand(CommandInterface $command)
    {
        try {
            $result = $this->execute($command);
            return $result->get('@metadata')['statusCode'] !== 404;
        } catch (S3Exception $e) {

This works for me locally for now, although I'm not sure whether this aligns with the SDK design philosophy.

[stobrien89]

Hi @manarth,

Could you describe the unexpected behavior that's occurring when you use doesObjectExistV2 ?

[manarth]

The same as above: checking for a non-existing key, in an existing bucket, returns TRUE – i.e. it believes the object exists.

$result = $s3Client->doesObjectExistV2("bucket-which-exists", "dsbhjbsdhjdbfjhs");
var_dump($result);
// bool(true)

[stobrien89]

Hi @manarth,

Thanks for confirming. I think it'd be helpful for us to take a look at the debug logs here— we definitely want to correct this if there's something wrong with the implementation. Could you post the S3 response from the debug logs after adding 'debug' => true to your client configuration? Please obscure any sensitive information such as keys or account numbers (if applicable). Thanks!

[github-actions]

This issue has not recieved a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.