S3 to GCS translation issues

Multipart Uploads

While GCS resumable uploads can be emulated without major problems using S3 multipart uploads, the opposite (emulating S3 multipart uploads using GCS resumable uploads) is not so straightforward.

The reason is that GCS resumable uploads have the following constraints:

• We must know the total size of the upload from the beginning, unlike S3 multipart. If we don't provide the total size, GCS API will assume it is a single chunk upload.

• The parts (or chunks) size must be a multiple of 256KiB (256*1024 bytes).

To overcome those limitations, Flasback nodes emulate a S3 multipart upload sequence by treating each part as separate files, and when the client calls CompleteMultipartUpload, it combines all files in the final destination file.