ascp: Transferring from the Command Line |
File checksums are useful for trouble-shooting file corruption, allowing you to determine at what point in the transfer file corruption occurred. Aspera servers can report source file checksums that are calculated on-the-fly during transfer and then sent from the source to the destination. To do so, the transfer must meet both of the following requirements:
The user on the destination can calculate a checksum for the received file and compare it (manually or programmatically) to the checksum reported by the sender. The checksum reported by the source can be retrieved in the destination logs, a manifest file, in Aspera Console, or as an environment variable. Instructions for comparing checksums follow the instructions for enabling checksum reporting.
Checksum reporting is disabled by default. You can enable and configure checksum reporting on the server by using the following methods:
Command-line options override the settings in aspera.conf and the GUI.
asconfigurator Option |
Description |
---|---|
file_checksum |
Enable checksum reporting and specify the type of checksum to calculate for
transferred files.
Note: The default value for the ascp option is
none, in which case the reported checksum is the
one configured on the server, if any.
|
file_manifest |
The file manifest is a file that contains a list of content that was
transferred in a transfer session. The file name of the file manifest is
automatically generated from the transfer session ID. When set to none, no file manifest is created. (Default) When set to text, a text file is generated that lists all files in each transfer session. |
file_manifest_path |
The location where manifest files are written. The location can be an
absolute path or a path relative to the transfer user's home directory. If
no path is specified (default), the file is generated under the destination
path at the receiver, and under the first source path at the sender. Note:
File manifests can be stored only locally. Thus, if you are using S3 or
other non-local storage, you must specify a local manifest path.
|
To enable checksum reporting, run the following asconfigurator command:
# asconfigurator -x "set_node_data;file_checksum,checksum"
To enable and configure the file manifest where checksum report data is stored, run the following commands:
# asconfigurator -x "set_node_data;file_manifest,text" # asconfigurator -x "set_node_data;file_manifest_path,filepath"
These commands create lines in aspera.conf as shown in the following example, where checksum type is md5, file manifest is enabled, and the path is /tmp.
<file_system> ... <file_checksum>md5</file_checksum> <file_manifest>text</file_manifest> <file_manifest_path>/tmp</file_manifest_path> ... </file_system>
To enable checksum reporting on a per-transfer-session basis, run ascp with the --file-checksum=hash option, where hash is sha1, md5, sha-512, sha-384, sha-256, or none (the default).
Enable the manifest with the option --file-manifest=output where output is either text or none. You can set the path to the manifest file with the option --file-manifest-path=path.
For example:
# ascp --file-checksum=md5 --file-manifest=text --file-manifest-path=/tmp file aspera_user_1@189.0.202.39:/destination_path
An alternative to enabling and configuring the file manifest to collect checksum reporting is to set up a pre/post-processing script to report the values.
The checksum of a transferred file is stored in the pre/post environment variable FILE_CSUM, which can be used in pre/post scripts to output file checksums. For example, the following script outputs the checksum to the file /tmp/cksum.log:
#!/bin/bash if [ $TYPE == File ]; then if [ $STARTSTOP == Stop ]; then echo "The file is: $FILE" >> /tmp/cksum.log echo "The file checksum is: $FILE_CSUM" >> /tmp/cksum.log chmod 777 $FILE fi fi
For information on pre- and post-processing scripts and environment variables, see IBM Aspera Enterprise Server Admin Guide: Testing and Optimizing Transfer Performance.
If you open a file that you downloaded with Aspera and find that it is corrupted, you can determine when the corruption occurred by comparing the checksum that is reported by Aspera to the checksums of the files on the destination and on the source.
$ md5 filepath
> CertUtil -hashfile filepath MD5
$ md5 filepath
# md5sum filepath
# csum -h MD5 filepath
# digest -a md5 -v filepath