Unfortunately, none of those mentioned above approaches solved our problem, so we came up with this approach. S3-dist-cp seems to be promising but when I ran it against a bucket with had closer to 6 TB of data the job failed while running “reduce” task after 40 minutes without any clear indication of why it failed Custom approach: When I read more about this AWS docs it stated under “Specifying a Manifest” section → Manifests that use server-side encryption with customer-provided keys (SSE-C) and server-side encryption with AWS KMS managed keys (SSE-KMS) are not supported Unsupported encryption type used: SSE_KMS
When I created a job to copy the contents of the bucket with KMS key encryption enabled got the following error: S3 batch operations seem to solve this problem, but at this point, it doesn’t support it on objects encrypted based on the KMS key. We tried a couple of other options mentioned in stack overflow and AWS forums like The only workaround we found is to run these aws commands in parallel in multiple terminals so they all can operate on different s3 partitions at the same time and perform copy faster, which is neither an elegant solution nor scalable.
During execution, we noticed it took hours and hours to perform the copy.
0 kommentar(er)
