AWS S3 Sync Issues #599

There have been a few issues with respect to the sync command, particularly in the case of syncing down from S3 (s3 -> local). I'd like to try to summarize the known issues as well as a few proposals of possible options, and give people the opportunity to share any feedback they might have.

Sync Behavior Overview

The sync behavior is intended to be an efficient cp; only copy over the files from the source to the destination that are different. In order to do that we need to be able to determine whether or not a file in s3/local are different. To do this, we use two values:

• File Size (from stat'ing the file locally and from the Size key in a ListObjects response)
• Last modified time (mtime of the local file and the LastModified key in a ListObjects response)

As an aside, we use the ListObjects operation because we get up to 1000 objects returned in a single call. This means that we're limited to information that comes back from a ListObjects response which is LastModified, ETag, StorageClass, Key, Owner, Size.

Now given the remote and local files file size and last modified times we try to determine if the file is different. The file size is easy, if the file sizes are different, then we know the files are different and we need to sync the file. However, last modified time is more interesting. While the mtime of the local file is a true mtime, the LastModified time from ListObjects is really the time the object was uploaded. So imagine this scenario:

aws s3 sync local/ s3://bucket/
sleep 10
aws s3 sync s3://bucket local/

After the first sync command (local->s3), the local files will have an mtime of 0, and the contents in s3 will have a LastModified time of 10 (using relative offsets). When we run the second aws s3 sync command, which is syncing from s3 to local we'll first do the file size check. In this case the file sizes are the same so we look at the last modified time checks. In this case they are different (local == 0, s3 == 10). If we were doing a strict equality comparison then, because the last modified times are different, we would unnecessarily sync the files from s3 to local. So we can say that if the file sizes are the same and the last modified time in s3 is greater (newer) than the local file, then we don't sync. This is the current behavior.

However, this creates a problem if the remote file is updated out of band (via the console or some other SDK) and the size remains the same. If we run aws s3 sync s3://bucket local/ we will not sync the remote file even though we're suppose to.

Potential Solutions

Below are potential solutions.

1. Change the time checks to be a strict equality comparison. If the times are different, we sync. This has the issue that aws s3 sync local s3://bucket && aws s3 s3://bucket local will unnecessarily sync files. However, when we download a file, we set the mtime of the file to match the LastModified time so if you were to run aws s3 sync s3://bucket local again, it would not sync any files.
2. Modify the local mtimes when files are uploaded to match the Last Modified time of S3. We'd then change the file time checks to a strict equal/not equal check. The only downside to this approach is whether or not people would expect us to mess with the mtime of their files (I would expect not).
3. Add custom metadata to each object. This could potentially add some combination of local mtime, md5 so we can also compare md5 checksums if we want. However, we would have to switch from ListObjects to a HeadObject for every request. This would require 1000 times more API calls, and would slow down the sync in the case of many small files. In the steady state case as long as we can HeadObjects faster that we can download/upload them there shouldn't be an issue. A downside to this approach is that other tools won't be adding this metadata, which would make interop with other tools harder (if an object is missing the metadata we'd likely just sync).
4. Keep a local ETag cache. We could associate the file name with an ETag/local mtime/last modified time/md5 of the file. On the plus side we still use ListObjects, so we get 1000 objects back at a time. Downside is that every client will have to keep a cache. This is also the most complex solution in terms of implementation.

If there any other potential solutions I've left out, please chime in.