asw s3 sync repeatedly uploads the same, unchanged files

[BLuFeNiX]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though the User Guide and the API reference
• I've searched for previous similar issues and didn't find any solution

Describe the bug
A certain folder in my S3-compatible storage is consistently detected as out of sync with my local source directory. ie: aws s3 sync /data/foo/ s3://my.bucket.tld/foo/ will always print dozens of lines like this, every time it is run.

upload: data/foo/bar/buzz/123.jpg to s3://my.bucket.tld/foo/bar/buzz/123.jpg

The files are identical, and I have confirmed so by hashing them locally, and then downloading a copy directly via aws s3 cp and hashing that too.

There is an interesting clue as to the nature of the bug: changing the path to be more specific will get rid of the behavior. For example, rather than running aws s3 sync /data/foo/ s3://my.bucket.tld/foo/, if you run aws s3 sync /data/foo/bar/buzz/ s3://my.bucket.tld/foo/bar/buzz/ it will show no changed files, and not upload anything. The directories are the same in both scenarios, but it seems that there is some sort of comparison against other objects in the bucket.

SDK version number
aws-cli/2.0.14 Python/3.7.3 Linux/4.19.107 botocore/2.0.0dev18

Platform/OS/Hardware/Device
Ubuntu 20.04 in a docker container

To Reproduce (observed behavior)
Unknown. It only happens with some files.

Expected behavior
Unchanged files will not be uploaded.

Logs/output
I do not wish to post full logs, but here is a relevant message:

2020-05-20 07:01:57,245 - MainThread - awscli.customizations.s3.syncstrategy.base - DEBUG - syncing: [REDACTED] -> [REDACTED], file does not exist at destination

It is the same message for every file, even though the file does exist in he destination bucket.

Additional context
It looks like this problem has existed since at least 2014, based on a similar report here:
https://forums.aws.amazon.com/thread.jspa?threadID=146851

I have also tried s3cmd, which does not exhibit this behavior.

[KaibaLopez]

Hi @BLuFeNiX ,
Sorry for the late response, I have not been able to reproduce this problem, is there anything you can think of that might be special about your case? any non-default bucket settings for example?

[albinj]

Same happens for me when I try to run s3 sync multiple times:
aws s3 sync "s3://static.domain.tld/" "folder"

It keeps downloading the same identical files. It only happens with files that have a special character in their filename. Such as ó æ á ð ø ú é í.

[agconti]

I have this same problem but I don't have special characters in my file names. The commands I'm using are:

aws s3 sync $BUILD_DIR $S3_BUCKET --include "*.html" \
--metadata-directive REPLACE --expires 2034-01-01T00:00:00Z --acl public-read \
--cache-control no-cache

aws s3 sync $BUILD_DIR $S3_BUCKET --exclude "*.html"  \
--metadata-directive REPLACE --expires 2034-01-01T00:00:00Z --acl public-read \
--cache-control max-age=31536000,public

A file like icon.svg will be always be reuploaded, despite having not changed.

[agconti]

I figured out what my problem was. sync decides what to sync based on the file's timestamp and size. My website build process was always updating the output files timestamps even if their contents did not change.

Adding the --size-only fixed my problem by ignoring timestamps and telling sync to update files only if their contents changed.

Hope this helps!

[BLuFeNiX]

I figured out what my problem was. sync decides what to sync based on the file's timestamp and size. My website build process was always updating the output files timestamps even if their contents did not change.

Adding the --size-only fixed my problem by ignoring timestamps and telling sync to update files only if their contents changed.

Hope this helps!

This is not the case with my situation. Files are identical (immutable, in fact) and still upload every time.

[boomshadow]

I figured out what my problem was. sync decides what to sync based on the file's timestamp and size. My website build process was always updating the output files timestamps even if their contents did not change.

Adding the --size-only fixed my problem by ignoring timestamps and telling sync to update files only if their contents changed.

Hope this helps!

This was exactly my problem. Thank you so much!

[agconti]

@boomshadow sure ting!

[kdaily]

Hi @BLuFeNiX,

Without seeing more of the logs this is difficult to diagnose. Do you happen to be using a third-party S3-compatible API (BackBlaze, DreamHost, etc). The symptom you're encountering sounds a lot like this: #5456 (comment)

In that case, the S3-compatible API was using the ListObjects implementation instead of ListObjects v2.

• https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects.html
• https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects-v2.html

This would explain why specifying a more specific prefix path (probably with less files, under 1000) would change the behavior.