In reading the documentation I'm concerned about the AWS sync logic - are you saying that the default behavior when syncing down from S3 is that an older S3 version will replace the newer local version?
The default behavior is to ignore same-sized items unless the local version is newer than the S3 version. I think I see what your'e getting at but by and large it seems confusing to have a sync logic in which an older file replaces a newer file. This is a really nasty bug and lead to an error in production that took me days to track down. Basic tools like this should just work without users having to know implementation details. Why doesn't sync use something sane like a checksum instead of a timestamp or file size?
It is and this is still a bug. The older file in S3 will overwrite the newer file regardless of whether or not the sizes match. To test run:. The file that was modified after the bucket file will then be overwritten and the modified date on the local file will now match the older S3 bucket file's modified date.
This change led to the same size property file, so the sync from the s3 bucket to the local machine failed to get the change. Obviously very different behavior. I'm being effected by this due to different timezones between the uploader and downloader. It seems that the explanation given in the first answer might be obsolete: when I sync a file from S3 to local, the local file then has the same last-modified time as shown on S3.
I need a strategy opposite to the default one: when syncing from S3 to local, if S3 is newer, update the local file. We are still experiencing this problem, a local file index. The sync command does not upload the file. I have the same problem. Need to sync data from local machine to S3. Request the ticket to be reopened. At the very least, awscli should make sure that the documentation is consistent with the code.
I ran into the same problem, and using "--exact-timestamps" option worked. As per documentation , the default behavior is to ignore same-sized items unless the local version is newer than the S3 version. I agree with checked answer. To add additional context, AWS S3 functionality is different than standard linux s3 in a number of ways.
In Linux, an md5hash can be computed to determine if a file has changed. What's worse, AWS does not preserve timestamp when transferring either way, so timestamp is ignored when syncing to local and only used when syncing to s3.
Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Collectives on Stack Overflow. Learn more. Ask Question. Asked 4 years, 7 months ago. Active 2 years, 10 months ago. Viewed 14k times. Can't figure out what might be happening. Any ideas? Costa Michailidis Costa Michailidis 6, 11 11 gold badges 61 61 silver badges bronze badges.
If you put a delay between the two executions, does it behave better? I tried with a few minutes apart. Same result. Add a comment. Active Oldest Votes. Naraen Naraen 3, 2 2 gold badges 19 19 silver badges 20 20 bronze badges.
There is an --exact-timestamps option where same-sized items will be ignored when the timestamps match exactly. The default behavior is to ignore same-sized items unless the local version is newer than the S3 version. How Does S3 Store Data? What Is S3? Amazon S3, aka S3 for short, is a…. Before implementing the command to rename the folders and….
How do I backup my S3 bucket data? How do I keep two S3 buckets in sync? Is it better to have multiple S3 buckets or one bucket with sub folders? Can S3 replicate to multiple buckets? If that is the truth, then why are folders noticeable in the S3 Console? This article answers all the in-depth questions you have about S3s, which create folders and which do not.
But before we jump into this topic, first things: Table of Contents. Previous Previous. Next Continue. Cloud Computing Expand child menu Expand. AWS Expand child menu Expand. General Tech Expand child menu Expand.
0コメント