← All posts

S3 said 403 Access Denied. The file simply was not there

Uploaded images returned 403 from a public bucket, and the first idea was to open access wider. The real cause was a key that did not exist: the SDK had put the bucket name into the object path.

A request came in from developers: images uploaded through the admin panel return 403 AccessDenied on their direct bucket link. The suggested fixes were the usual ones — open the bucket, or let traffic in from the VPN.

Neither would have helped. The bucket was fine; the objects were not where the links pointed.

Why S3 says 403 for a missing object

If the caller does not have s3:ListBucket, S3 answers a request for a non-existent key with 403, not 404. This is deliberate: otherwise anyone could probe a bucket and learn which object names exist.

Anonymous requests almost never have s3:ListBucket. So for a public link, 403 means either “no permission” or “no such object”, and from the outside you cannot tell them apart.

There is no network meaning in that code at all. The public S3 endpoint is reachable from anywhere, a VPN changes nothing. If you got a proper XML body with <Code>AccessDenied</Code>, the request reached S3. An unreachable endpoint looks like a timeout or a refused connection, not like a polite error document.

The check that decides it

Three steps, and the third one is the answer:

# 1. does the object exist (needs credentials)
aws s3api head-object --bucket my-bucket --key 'images/photo.jpg'

# 2. what the bucket policy and public access block say
aws s3api get-bucket-policy --bucket my-bucket --query Policy --output text
aws s3api get-public-access-block --bucket my-bucket

# 3. control: a file that certainly exists, same prefix, same anonymous request
curl -s -o /dev/null -w '%{http_code}\n' https://my-bucket.s3.<region>.amazonaws.com/images/known.jpg

200 on the neighbour and 403 on the file in question means the permissions are fine and the key is missing.

The bucket name inside the key

Listing the bucket showed a top-level prefix with the same name as the bucket itself. The new uploads were stored as my-bucket/images/photo.jpg inside my-bucket.

The cause was one configuration value. https://my-bucket.s3.<region>.amazonaws.com is a virtual-hosted-style address: the bucket name is already in the hostname. The application passed it to the SDK as the endpoint, the SDK worked in path-style and added the bucket name once more — this time as the first segment of the key.

For the SDK the endpoint must be the regional one without the bucket, https://s3.<region>.amazonaws.com. The public base URL used to build browser links keeps the bucket in it. Two parameters with almost identical values, and it is easy to put one into the other.

A useful detail was the dates. Several hundred objects under correct paths were old, from the original migration. Only two objects sat under the duplicated prefix, both from that same day. The upload had broken recently, it had not always been like this — which pointed at a fresh config change rather than at the bucket.

What not to do

Do not open the bucket wider “just in case” before step 3. If the key is wrong, wider access fixes nothing, and the extra exposure stays forever because nobody remembers why it was added.

A CDN in front of the bucket does not help either: it will faithfully serve the same missing object.

Takeaways

  • From S3, an anonymous 403 is ambiguous. Treat it as “permission or missing key” until proven otherwise.
  • Always compare with a known-good object under the same prefix and the same kind of request.
  • An XML error body means the network is fine. Stop looking at VPNs and firewalls.
  • SDK endpoint and public base URL are different settings. The SDK gets the regional endpoint without the bucket.
  • If a bucket contains a prefix equal to its own name, you have found the bug.