Cloud Transferors

We provide a helper method to transfer S3 data to an azure blob.

Warning

this method should only be used for < 1 Tb content, as it is quite slow. Look at azcopy for larger transfers.

def transfer_s3_to_blob(folders: list[Path],
                        index_folder: Path,
                        bucket: str = BUCKET,
                        container: str = CONTAINER
                        ) -> None:

Attributes are:

• folders: the list pathlib Path specifying where the data to transfer from the s3 bucket to the blob container are located. Remember to read the forge key page to learn how to properly... Well, forge a key 😊.
• index_folder: where to store index files (in order not to re-transfer already successfully transferred files, and to store files for which transfer failed)
• bucket (optional): the S3 bucket name from which to transfer. By default, the s3_bucket_name environment variable is used.
• container (optional): the AWS container name to which to transfer. By default, the container environment variable is used.