AWS S3 sync does not sync all the files

[webdigi]

We have several hundred thousand files and S3 reliably syncs files. However, we have noticed that there were several files which were changed about a year ago and those are different but do not sync or update.

Both source and destination timestamps are also different but the sync never happens. S3 has the more recent file.

Command is as follows
aws s3 s3://source /local-folder --delete

All the files that do not sync have the same date but are spread across multiple different folders.

Is there an S3 touch command to change the timestamp and possibly get the files to sync again?

[JordonPhillips]

You can possibly use --exact-timestamps to work around this, though that may result in excess uploads if you're uploading.

To help in reproducing, could you get me some information about one of the files that isn't syncing?

• What is the exact file size locally?
• What is the exact file size in S3?
• What is the last modified time locally?
• What is the last modified time in S3?
• Is the local file a symlink / behind a symlink?

[webdigi]

Example command run
aws s3 sync s3://bucket/ /var/www/folder/ --delete

Several files are missing
Exact local size: 2625
Exact s3: 2625
Exact time stamp local: 06-Jan-2017 9:32:31
Exact time stamp s3: 20-Jun-2017 10:14:57
normal file in S3 and local

There are several cases like that in a list of around 50,000 files. However all the missing in sync are various times on 20 Jun 2017.

Using --exact-timestamps shows much more files to download although they are exactly the same contents. However they are still missing the ones in example above.

[overcache]

same issue here.
aws s3 sync dist/ s3://bucket --delete did not upload s3://bucket/index.html with dist/index.html

dist/index.html and s3://bucket/index.html have the same file size, but their modify time are different.

actually, some times awscli did upload the file, but some times not

[zyv]

Same here, --exact-timestamps doesn't help - index.html is not overwritten.

[samdammers]

We experienced this issue was well today/last week. Again index.html is the same file size, but the contents and modified times are different.

[stephram]

Is anybody aware of a workaround for this?

[lylejohnson]

I just ran into this. Same problem as reported by @icymind and @samdammers: the contents of my (local) index.html file had changed, but its file size was the same as the earlier copy in S3. The {{aws s3 sync}} command didn't upload it. My "workaround" was to delete index.html from S3, and then run the sync again (which then uploaded it as if it were a new file, I guess).

[smxdevst]

Server: EC2 linux
Version: aws-cli/1.16.108 Python/2.7.15 Linux/4.9.62-21.56.amzn1.x86_64 botocore/1.12.98

---

After aws s3 sync running over 270T of data I lost few GB of files. Sync didn't copy files with special characters at all.

Example of file /data/company/storage/projects/1013815/3.Company Estimates/B. Estimates

Had to use cp -R -n

[checkmypi]

same issue here xml file of the same size but different timestamp not synced correctly

I was able to reproduce this issue

bug.tar.gz
download attached tar file and then

tar -zxvf bug.tar.gz
aws s3 sync a/ s3://<some-bucket-name>/<some_dir>/ --delete
aws s3 sync b/ s3://<some-bucket-name>/<some_dir>/ --delete

you'll see that even though repomd.xml in directories a and b differ in contents and timestamps
attempting to sync b doesn't do anything

Tested on
aws-cli/1.16.88 Python/2.7.15 Darwin/16.7.0 botocore/1.12.78
aws-cli/1.16.109 Python/2.7.5 Linux/3.10.0-693.17.1.el7.x86_64 botocore/1.12.99

[chrispruitt]

im seeing the same issue. trying to sync a directory of files from s3 where one file was updated to a local directory. that file does not get updated in the local directory

[lqueryvg]

I'm seeing this too. In my case it's a react app with index.html that refers to generated .js files. I'm syncing them with the --delete option to delete old files which are no longer referred to. The index.html is sometimes not uploaded, resulting in an old index.html which points to .js files which no longer exist.

Hence my website stops working !!!

I'm currently clueless as to why this is happening.

Does anyone have any ideas or workarounds ?

[marns93]

We have the same problem, but just found a workaround. I know, it is not the best way, but it works:

aws s3 cp s3://SRC s3://DEST ...
aws s3 sync s3://SRC s3://DEST ... --delete

It seems to us, that the copy is working fine, so first we copy after that we use the sync command to delete files, which are no longer present.
Hope that the issue will be fixed asap.