Restrictions on file copies

In Flashback Network, a repository can connect to several different buckets, with the same or different storage type and providers.

There are 2 different ways to handle file copy operations between buckets:

1

Using the provider's native copy function

Native copy functions are

• S3: CopyObject

• GCS: RewriteObject

• Azure blob: blob.copy_from_url

Native copy can only work if:

• The source and destination storage types and providers are the same.

• Both source and destination buckets abide by the conditions the provider applies for a native Copy function to be allowed. For example, some providers may restrict native copies between buckets, or may allow them but with extra latency and chages, depending on:

  • the buckets being in different regions.

  • or in different storage classes (hot vs warm vs cold)

  • or the file being above a certain size.

  • others.

Check out in S3 docs, GCS docs and Azure Blob docs for more detail on platform specific conditions and constraints.

How to signal Flashback nodes that we want to attempt a native copy function:

• When connecting to a repository with API type S3 or Azure Blob

  • The destination bucket must be associated to the repo

  • The source bucket cannot be associated to the repo.

  • The credentials for the destination bucket must grant access to the source bucket as well.

• When connecting to a repository with GCS:

  • The source bucket must be associated to the repo

  • The destination bucket cannot be associated to the repo.

  • The credentials for the source bucket must grant access to the destination bucket as well.

It's the responsibility of the user to check both the provider's and Flashback requisites to decide attempting a native copy operation.

In case of doubt, before starting one or more big file copy operations, you can try with a small one and then check in the provider's console if the test copy caused a charge in traffic costs.

2

Emulating the copy function

If the source and destination buckets belong to different storage sources or providers, or if they belong to the same one, but the native copy is not supported, Flashback nodes will try to "emulate" the copy operation.

How to signal Flashback nodes that we want to attempt an emulated copy operation:

• Both source and destination buckets must be associated to the repo we are connecting to.

• Credentials for each bucket can be the same or different (that makes no difference)

The main differences from case 1 and case 2 are:

• with native copy, we can save in traffic charges if we follow all the requirements from the provider, and the file size transferred will not count towards the monthly spending in our Flashback Network account.

• with emulated copy, we will always pay for the outgoing traffic from the source bucket, and the file size transferred will always count towards the monthly spending in our Flashback Network account.