`aws s3 sync` downloading unchanged files.

[MrJoy]

Describe the bug

I have a maintenance script I run to keep a local copy of billing & usage data for my personal AWS account. It's identifying almost every file as changed, on every run even though most of the files haven't been modified in years.

Expected Behavior

Only changed files -- in this case, files representing the current billing period -- should be downloaded.

Current Behavior

Of 6,279 files that do not represent the current billing period, it's consistently re-downloading 5,831 of them. The files it downloads are, byte-for-byte identical to the existing ones. I spot-checked one of the files, and aws s3 ls reports the exact same size and timestamp as ls does.

Reported by aws s3 sync:

download: s3://mrjoy-billing-data//cur//billing_and_usage/20210101-20210201/20210122T235314Z/billing_and_usage-00001.csv.gz to ../personal/Finance/AWS_Billing_Data/cur/billing_and_usage/20210101-20210201/20210122T235314Z/billing_and_usage-00001.csv.gz

Reported by aws s3 ls:

% aws-vault exec mrjoy -- aws s3 ls s3://mrjoy-billing-data//cur//billing_and_usage/20210101-20210201/20210122T235314Z/billing_and_usage-00001.csv.gz
2021-01-22 15:53:24     296522 billing_and_usage-00001.csv.gz

Reported by ls:

% ls -laD "%Y-%m-%d %H:%M:%S" ~/personal/Finance/AWS_Billing_Data/cur/billing_and_usage/20210101-20210201/20210122T235314Z/billing_and_usage-00001.csv.gz
-rw-r--r--  1 jonathonfrisby  staff  296522 2021-01-22 15:53:24 /Users/jonathonfrisby/personal/Finance/AWS_Billing_Data/cur/billing_and_usage/20210101-20210201/20210122T235314Z/billing_and_usage-00001.csv.gz

The post-fetch commit in all cases shows diffs for the files in the current billing period (as would be expected), and no changes to any of the other files that aws s3 sync reports as being downloaded.

All told, aws s3 sync appears to be downloading around 700MB of files on each run that it shouldn't be.

Reproduction Steps

The relevant portion of my script is:

#!/bin/bash
IFS=$'\n\t'
set -euo pipefail

(
  cd ~/personal
  git add .
  git commit --all --allow-empty -m "AWS bill snapshot, pre-fetch..."
  aws-vault exec mrjoy -- aws s3 sync s3://mrjoy-billing-data/ ~/personal/Finance/AWS_Billing_Data/
  git add .
  git commit --all --allow-empty -m "AWS bill snapshot, post-fetch..."
)

The data in the bucket is written by AWS itself.

Possible Solution

No response

Additional Information/Context

No response

CLI version used

2.7.26

Environment details (OS name and version, etc.)

macOS 12.5.1

[tim-finnigan]

Hi @MrJoy thanks for reaching out. Have you tried using the --size-only parameter documented here? This parameter makes the size of each key the only criteria used to decide whether to sync from source to destination. So it should ignore all of those files that are the same size.

[MrJoy]

I attempted it just now, and it did not change anything. It's still attempting to download ~everything. Note that after submitting this ticket, I updated to 2.7.27 -- so this test was done on 2.7.27 not 2.7.26.

[tim-finnigan]

Thanks for the update. There is an older issue tracking problems with S3 sync here: #599. Some users have reported anomalies when certain files sync that should not, but I wouldn't expect the problem at the scale you're describing where it's happening with hundreds of files. I don't know if I'd be able to reproduce the issue as described but could try. If you can get the debug logs by adding --debug to the command that might also give more insight into the problem. Some have said that using --size-only or --exact-timestamps has helped produce the expected results. There are other S3 sync-related feature requests like #6750 that relate to using new checksum algorithms for improving the accuracy.

[MrJoy]

@tim-finnigan I'm sorry, I was unclear in my last message: When I said "I attempted it just now", I meant "I attempted to use --size-only just now". Accidental pronoun game, FTL.

If you'd like, I can temporarily give you read-only credentials for this bucket and you can see if you are able to recreate the problem from the same source. My personal AWS bill is... not data I'm terribly worried about sharing.

I'll get --debug output and add it here tomorrow. I'm about to be AFK for a while.

[MrJoy]

debug.log.cleansed.zip

I've stripped tokens/signatures/key IDs from the file, but it's otherwise as produced from running:

aws-vault exec mrjoy -- aws s3 sync --debug --size-only s3://mrjoy-billing-data/ ~/personal/Finance/AWS_Billing_Data/

[tim-finnigan]

Thanks @MrJoy for following up and sharing your logs. I couldn't identify any anomalies after scanning through the logs. I think attempting to recreate the issue is a good idea, but for that I recommend reaching out through AWS Support to open a private communication channel. I'd also recommend trying to use --exact-timestamps when running the sync command to see if that addresses the issue you're seeing.

[MrJoy]

I went ahead and tried --exact-timestamps by itself and in combination with --size-only, and the behavior seems to be the same in all cases.

Going through AWS Support is not an option, as this is my personal account and I'm on the Basic plan.

[tim-finnigan]

Checking in on this issue again - thanks for your patience. I think this issue might actually overlap with #5730, #648 and/or #5369. Have you looked through any of those issues? Based on some of the comments it sounds like this could be due to how S3 handles timestamps.