Parallel Transfers to Amazon S3 and Amazon EC2

Parallel transfers to cloud storage allow faster uploads by splitting large sets of files across multiple ascp sessions. They are currently supported only for command-line transfers to Amazon S3 and Amazon EC2.

Note: If you are uploading to a cluster, you must use access keys for authentication rather than IAM roles to run parallel transfers, unless you are using Assumed roles.

Server Configuration to Support Parallel Transfers

SSH into your Aspera server:

# ssh -i identity_file -p 33001 ec2-user@ec2_host_ip
#  su -

Set a value for the chunk size that is greater than 5 Mb by running the following command (this example sets the chunk size to 64 Mb):

# asconfigurator -x "set_node_data;transfer_protocol_options_chunk_size,67108864"

Open /opt/aspera/etc/trapd/s3.properties. Set the upload part-size to the same value as the chunk size (the default is 64 Mb) and set the gathering policy to ONE_TO_ONE, as in the following example:

aspera.transfer.upload.part-size=64MB
aspera.transfer.gathering-policy=ONE_TO_ONE

Ascp Command Syntax

ascp --tags={\"aspera\":{\"xfer_id\":\"transfer_id\"}} -C nid:ncount [[user@]srcHost:]source_file1[,source_file2,...] [[user@]destHost:]target_path

Notes on the command syntax:

  • The xfer_id tag is required and identifies the upload across multiple ascp sessions. If an upload is restarted with the same xfer_id then the transfer is resumed, but if a different xfer_id is used then the upload is completely restarted.
  • The -C nid:ncount option is used to specify the node ID (nid) and count (ncount) in the format 1:2, 2:2. The valid range of values for nid and ncount is 1 through 128, and nid must be less than or equal to ncount.

Parallel transfer behavior:

  • The ascp session for the first chunk creates the AWS upload and the ascp sessions for the other chunks might wait until the AWS upload exists.
  • When a chunk is uploaded, the ascp session evaluates if all parallel transfers are complete. If more than four chunks are pending, the chunk is finished and the ascp session terminates. If four or fewer chunks are pending, the chunk waits for the remaining chunks to upload, then the upload is finalized.
  • If the source content is smaller than the chunk size, only one ascp session is launched.