Ascp Transfers with Object Storage and HDFS

With an Aspera On Demand-entitled Aspera server installed in your cloud or on-premises object storage, you can use ascp to transfer to and from it. The syntax of an ascp command transferring to cloud or on-premises object storage depends on how you authenticate the transfer. The following options for authenticating to the object storage are described below:

Authenticating the Aspera Transfer User

You must enter the transfer user's password each time you run an ascp transfer, unless you either set the transfer user's password as an environment variable or set up an SSH key (token) and specify it in the command.

With Docroot Configured: Authenticate in the Docroot

If your transfer user account has a docroot set, ascp transfers to and from AWS S3, IBM COS - S3, Google Cloud Storage, Akamai, Softlayer, and Azure are the same as regular ascp transfers. For command syntax examples, see Ascp General Examples.

For instructions on configuring a docroot for these types of storage, see Aspera Enterprise Server Admin Guide (Linux): Docroot Path Formatting for Cloud, Object, and HDFS Storage. You are prompted for the transfer user's password upon running these commands unless you have set the ASPERA_SCP_PASS environment variable or are using an SSH key, as described previously.

With No Docroot Configured: Authenticate with Environment Variables

You can set an environment variable (ASPERA_DEST_PASS) with the storage password or access key using the command below:

> set ASPERA_DEST_PASS = secret_key

With this and ASPERA_SCP_PASS set, run ascp with the syntax listed in the table above, but you do not need to include the storage password or access key, and are not prompted for the Aspera password upon running the command.

Note: The ASPERA_DEST_PASS variable is not applicable to Google Cloud Storage or AWS S3 using IAM roles.

With No Docroot Configured: Authenticate in the Command Line

If you do not have a docroot configured and do not set an environment variable (described previously), you must authenticate in the command line. In the examples below, you include the storage password or secret key as part of the destination path. You are prompted for the transfer user's password upon running these commands unless you have set the ASPERA_SCP_PASS environment variable or are using an SSH key, as described above.

Storage Platform ascp Syntax and Examples
AWS S3
  • If you are using IAM roles, you do not need to specify the access ID or secret key for your S3 storage.
Upload syntax:
> ascp options --mode=send --user=username --host=s3_server_addr source_files s3://access_id:secret_key@s3.amazonaws.com/my_bucket

Upload example:

> ascp --mode=send --user=bear --host=s3.asperasoft.com bigfile.txt s3://1K3C18FBWF9902:GEyU...AqXuxtTVHWtc@s3.amazonaws.com/demos2014

Dowload syntax:

> ascp options --mode=recv --user=username --host=s3_server_addr s3://access_id:secret_key@s3.amazonaws.com/my_bucket/my_source_path destination_path

Download example:

> ascp --mode=recv --user=bear --host=s3.asperasoft.com s3://1K3C18FBWF9902:GEyU...AqXuxtTVHWtc@s3.amazonaws.com/demos2014/bigfile.txt /tmp/
Azure These examples are for Azure blob storage. For Azure Files, use the syntax: azure-files://storage_account:storage_access_key@file.core.windows.net/share

Upload syntax:

> ascp options --mode=send --user=username --host=server_address source_files azu://storage_account:storage_access_key@blob.core.windows.net/path_to_blob

Upload example:

> ascp --mode=send --user=AS037d8eda429737d6 --host=dev920350144d2.azure.asperaondemand.com bigfile.txt azu://astransfer:zNfMtU...nBTkhB@blob.core.windows.net/abc  

Dowload syntax:

> ascp options --mode=recv --user=username --host=server azu://storage_account:storage_access_key@blob.core.windows.net/path_to_blob/source_file destination_path

Download example:

> ascp --mode=recv --user=AS037d8eda429737d6 --host=dev920350144d2.azure.asperaondemand.com azu://astransfer:zNfMtU...nBTkhB@blob.core.windows.net/abc /downloads
Google Cloud Storage Note: The examples below require that the VMI running the Aspera server is a Google Compute instance.
> ascp options --mode=send --user=username --host=server_address source_files gs:///my_bucket/my_path

Upload example:

> ascp --mode=send --user=bear --host=10.0.0.5 bigfile.txt gs:///2017_transfers/data  

Dowload syntax:

> ascp options --mode=recv --user=username --host=server gs:///my_bucket/my_path/source_file destination_path

Download example:

> ascp --mode=recv --user=bear --host=10.0.0.5 gs:///2017_transfers/data/bigfile.txt /data
HDFS Aspera recommends running ascp transfers with HDFS with a docroot configured.
IBM COS - S3 Upload syntax:
> ascp options --mode=send --user=username --host=server_address source_files s3://access_id:secret_key@accessor_endpoint/vault_name

Upload example:

> ascp --mode=send --user=bear --host=s3.asperasoft.com bigfile.txt s3://3ITI3OIUFEH233:KrcEW...AIuwQ@38.123.76.24/demo2017

Dowload syntax:

> ascp options --mode=send --user=username --host=server_address s3://access_id:secret_key@accessor_endpoint/vault_name/source_files destination_path

Download example:

> ascp --mode=send --user=bear --host=s3.asperasoft.com s3://3ITI3OIUFEH233:KrcEW...AIuwQ@38.123.76.24/demo2017 /tmp/
IBM Cloud Object Storage (COS) - Swift and IBM Bluemix Aspera recommends running ascp transfers with IBM Cloud Object Storage (COS) - Swift and IBM Bluemix with a docroot configured.
OpenStack Swift Upload syntax:
> ascp options --mode=send --user=username --host=ip_addr source_files swift://account_id:api_key@auth_url/my_bucket

Example Upload:

> ascp --mode=send --user=bear --host=192.155.218.130 bigfile.txt swift://XYZO...46-2:bob:437e...bc16@sjc01.objectstorage.service.networklayer.com/test

Dowload syntax:

> ascp options --mode=recv --user=username --host=ip_addr swift://account_id:api_key@auth_url/my_bucket/my_source_path destination_path

Download example:

> ascp --mode=recv --user=bear --host=192.155.218.130 swift://XYZO...46-2:bob:437e29...f616@sjc01.objectstorage.service.networklayer.com/test/bigfile.txt /tmp/
Note: Swift requires additional Trapd configuration settings that can be included as queries attached to the docroot, with the format docroot?setting.

For example, for an upload to IBM COS - Swift, the path is written as follows:

swift://XYZO...46-2:bob:437e...bc16@sjc01.objectstorage.service.networklayer.com/test?aspera.swift.endpoint.auth-path=/auth/v1.0