Multi-Session Transfers

Ascp can transfer content faster by using multi-session transfers (also known as parallel transfers and multi-part transfers) to and from multi-node and multi-core servers and clusters, on premises or in the cloud. This article describes the syntax of a multi-session transfer and provides an example.

Multi-session syntax

To run simultaneous ascp transfers, you can run each command from its own terminal window, run a script in a single terminal, or background processes with the shell.

For a typical push (--mode=send) transfer:

# ascp -C nid_1:ncount -l max_rate [-O port_1][--multi-session-threashold=threshold] [--tags={\"aspera\":{\"xfer_id\":\"transfer_id\"}}] source_path hostname:destination_path
# ascp -C nid_2:ncount -l max_rate [-O port_2] [--multi-session-threashold=threshold] [--tags={\"aspera\":{\"xfer_id\":\"transfer_id\"}}] source_path hostname:destination_path
...
# ascp -C nid_n:ncount -l max_rate [-O port_n] [--multi-session-threashold=threshold] [--tags={\"aspera\":{\"xfer_id\":\"transfer_id\"}}] source_path hostname:destination_path

Where:

  • -C nid:ncount tells Ascp that the same source and destination are being used by multiple, concurrent sessions. nid is the node ID and ncount is the number of nodes or cores. The valid range of values for nid and ncount is 1 through 128, and nid must be less than or equal to ncount.
  • -O port is used to assign each session to a different UDP port. This is required when the server's operating system does not support concurrent sessions using the same UDP port. This applies to Windows, macOS, Isilon OneFS, and Solaris operating systems.
    Note: Make sure that the server's firewall is configured to accept transfers on the range of UDP ports.
  • --multi-session-threshold is an optional argument that enables files to be split between sessions. The threshold value specifies, in bytes, the smallest-size file that can be split. Files greater than or equal to the threshold are split, while those smaller than the threshold are not. If the multi-session threshold is set to 0 (zero), files are not split.
  • --tags={\"aspera\":{\"xfer_id\":\"transfer_id\"}} is required for multi-session transfers to cloud in order to provide a transfer ID. The transfer ID is the same for all the sessions in the multi-session transfer. If an upload is restarted with the same xfer_id then the transfer is resumed, but if a different xfer_id is used then the upload is completely restarted.
  • If you are uploading to a cloud-based AWS S3 cluster, you must authenticate with an access key or Assumed role rather than an IAM role.
  • If you are self-managing an Aspera server or cluster of Aspera servers in the cloud (you installed IBM Aspera High-Speed Transfer Server on a VM), you must configure the server for multi-session transfers.

File-spliting with multi-session threshold

The value of the multi-session threshold depends on the target rates that a single ascp transfer can achieve on your system for files of a given size, as well as the typical distribution of file sizes in the transfer list.

Note: A default value for the threshold can be specified in the server and client aspera.conf by setting <multi-session_threshold_default> in the <default> section. The command-line setting overrides the aspera.conf setting. If the client's aspera.conf does not specify a default value for the threshold, then the server's setting is used (if specified). If neither the client nor the server set a multi-session threshold, then no files are split.

To set a value (in bytes) from the command line, run the following: 

# asconfigurator -x "set_node_data;transfer_multi_session_threshold_default,threshold"

Multi-Session Transfer Example

The following example shows a multi-session transfer on a dual-core system. Together, the two sessions can transfer at up to 2 Gbps and each session uses a different UDP port. No multi-session threshold is specified on the command line or in aspera.conf, so no file splitting occurs.

# ascp -C 1:2 -O 33001 -l 1000m /dir01 10.0.0.2:/remote_dir
# ascp -C 2:2 -O 33002 -l 1000m /dir01 10.0.0.2:/remote_dir

If dir01 contains multiple files, ascp distributes the files between each command to get the most efficient throughput. If dir01 contains only one file, only one of the commands transfers the file.

In the following example, the multi-session threshold is used to enable file splitting: 

# ascp -C 1:2 -O 33001 -l 100m --multi-session-threshold=5242880 /dir01 10.0.0.2:/remote_dir
# ascp -C 2:2 -O 33002 -l 100m --multi-session-threshold=5242880 /dir01 10.0.0.2:/remote_dir

In this case, if dir01 contains multiple files, all files less than 5 MB are distributed between sessions, while all files 5 MB or larger are split and then distributed between sessions. If dir01 contains only one file and that file is 5 MB or larger, then the file is split, otherwise the file is transferred by one session.