Cross-cloud migration and vendor-lock-in escape

Expiremental Guide: May contain errors as our technology continues to evolve. If you encounter any problems, please do not hesitate to contact us in Discord and give us your feedback.

The Problem

Moving data between clouds (S3 ⇄ GCS ⇄ Azure Blob) is painful: every provider has different copy semantics, auth models, and hidden limits. Refactors are common, and you often pay egress plus suffer long cut-overs.

Flashback lets you keep one client integration while using two migration paths: a provider’s native copy (S3 CopyObject, GCS RewriteObject, Azure copy_from_url) when rules allow, or an emulated copy streamed by Bridge Nodes when native isn’t possible. This unifies the workflow and makes it easy to test (which path you’re taking) and what it will cost.

---

Step-by-Step Deployment Recommendations

1

Model the migration policy

• Prefer native copy when the provider allows it (same storage type/provider, and the provider’s region/class/size rules). Otherwise Flashback emulates the copy by streaming, which incurs egress and counts toward Flashback traffic. Always test with a small object first to confirm cost behavior.

Signaling which path to use (critical):

• S3/Blob native copy: connect to the Repo; destination bucket must be attached to the Repo; the source bucket must not; destination credentials must also allow reading the source bucket.

• GCS native copy: inverse, source attached, destination not; source credentials must allow writing to destination.

• Emulated copy: attach both buckets to the same Repo; credentials may differ.

2

Create Buckets & Repository in Flashback

• Create/Link Cloud Buckets in Flashback

  • For each provider, add a Bucket in the dashboard: “Add Bucket” → select provider → provide credentials/role → validate.

• Create a Repository

  • “New Repository” → name it (e.g., app-data) → attach all created Buckets.

• Generate Repository API Keys

  • Create a WRITE key for your application and a READ key for services that only read.

  • Save the secret in your vault; you can’t retrieve it later from it.

3

Wire your backend client to Flashback

Create clients pointing at a Bridge Node endpoint; keep a small cache keyed by endpoint so you can swap quickly.

Python (boto3 / S3)

# fb_s3_client.py
import boto3
from botocore.client import Config

def s3_client_for(endpoint, key_id, key_secret):
    session = boto3.session.Session(
        aws_access_key_id=key_id, aws_secret_access_key=key_secret
    )
    return session.client("s3", endpoint_url=endpoint, config=Config(signature_version="s3v4"))

Node (aws-sdk v3 / S3)

// fbS3Client.ts
import { S3Client } from "@aws-sdk/client-s3";

const clients = new Map<string, S3Client>();
export function s3ClientFor(endpoint: string, keyId: string, secret: string) {
  if (!clients.has(endpoint)) {
    clients.set(endpoint, new S3Client({
      endpoint, region: "us-east-1",
      credentials: { accessKeyId: keyId, secretAccessKey: secret },
      forcePathStyle: true
    }));
  }
  return clients.get(endpoint)!;
}

(Use analogous clients for GCS/Azure if that’s your app’s native protocol.) Flashback S3 endpoints are compatible with standard S3 SDKs.

4

Implement the migration routine

A. Native copy (fast path, usually lowest cost)

Requirements recap: same provider/type; signal native per step 1; ensure the required cross-bucket permissions. Operations are the platform’s built-ins: S3 CopyObject, GCS RewriteObject, Azure copy_from_url.

Python (boto3 / S3 CopyObject)

from config_flashback_s3 import s3

def copy_native_s3(src_bucket: str, src_key: str, dst_bucket: str, dst_key: str):
  s3.copy_object(
      Bucket=dst_bucket,
      Key=dst_key,
      CopySource=f"{src_bucket}/{src_key}",  # URL-encode if needed
  )

Node (aws-sdk v3 / S3 CopyObject)

import { CopyObjectCommand } from "@aws-sdk/client-s3";
import { s3 } from "./configFlashbackS3";

export async function copyNativeS3(srcBucket: string, srcKey: string, dstBucket: string, dstKey: string) {
  await s3.send(new CopyObjectCommand({
    Bucket: dstBucket,
    Key: dstKey,
    CopySource: `${srcBucket}/${encodeURIComponent(srcKey)}`
  }));
}

B. Emulated copy (cross-provider or when native is disallowed)

Attach both buckets to the Repo; invoke the same copy operation. Flashback will stream the bytes between buckets. Expect outgoing traffic charges and count toward Flashback spend.

Large files: default single-upload limit is 100 MB; use multipart/resumable uploads for larger objects. (Multipart copy via UploadPartCopy is not supported; fall back to a read-then-multipart-put if you need client-side control.)

5

Configure quotas & alerts

• Quotas: when exceeded, repository write operations are disabled, reads remain available; surface QUOTA_EXCEEDED to pause jobs.

• Observability: pull daily/minute stats and node minute stats to watch throughput/latency and detect hotspots.

const H = { Accept: "application/json", Authorization: `Bearer ${process.env.FB_JWT}` };
const daily  = await fetch("https://backend.flashback.tech/stats/daily",  { headers: H }).then(r=>r.json());
const minute = await fetch("https://backend.flashback.tech/stats/minute", { headers: H }).then(r=>r.json());

6

(Optional) Add a cold tier

Attach a second (cheaper) destination bucket to the Repo and schedule periodic copies from your hot bucket. Prefer native copy when available to avoid traffic charges; otherwise emulated copy will stream and bill egress.

7

Validate & roll out

• Dry-run: copy a tiny object and check the cloud console for whether traffic was billed (confirms native vs emulated).

• Performance: pick the closest Bridge Node; bucket details expose Online/Disconnected/Offline and latencies measured via HeadBucket.

• Gradual rollout: start with low-risk prefixes, then scale up.

8

Operation Playbook

• Keys: rotate repo keys; secrets are non-recoverable after you leave the page.

• Quotas: treat QUOTA_EXCEEDED as read-only; resume only after reset or after raising limits.

• Retry/fallbacks: if native copy fails with provider rules, retry as emulated by attaching both buckets.

• Limits: >100 MB needs multipart/resumable; plan client-side chunking for huge objects.