Reporting Checksums
File checksums are useful for trouble-shooting file corruption, allowing you to determine at what point in the transfer file corruption occurred. Aspera servers can report source file checksums that are calculated on-the-fly during transfer and then sent from the source to the destination. To do so, the transfer must meet both of the following requirements:
- Both the server and client computers must be running Enterprise Server, Connect Server, or Point-to-Point Client version 3.4.2 or higher.
- The transfer must be encrypted. Encryption is enabled by default.
The user on the destination can calculate a checksum for the received file and compare it (manually or programmatically) to the checksum reported by the sender. The checksum reported by the source can be retrieved in the destination logs, a manifest file, in Aspera Console, or as an environment variable. Instructions for comparing checksums follow the instructions for enabling checksum reporting.
Checksum reporting is disabled by default. Enable and configure checksum reporting on the server by using the following methods:
- Edit aspera.conf with asconfigurator.
- Set options in the client GUI.
- Set ascp command-line options (per-transfer configuration).
Command-line options override the settings in aspera.conf and the GUI.
Overview of Checksum Configuration Options
|
Description |
---|---|
|
Enable checksum reporting and specify the type of checksum to calculate for
transferred files.
Note: The default value for the ascp option is
none , in which case the reported checksum is the
one configured on the server, if any. |
|
The file manifest is a file that contains a list of content that was
transferred in a transfer session. The file name of the file manifest is
automatically generated from the transfer session ID. When set to
When set to |
|
The location where manifest files are written. The location can be an
absolute path or a path relative to the transfer user's home directory. If
no path is specified (default), the file is generated under the destination
path at the receiver, and under the first source path at the sender. Note:
File manifests can be stored only locally. Thus, if you are using S3 or
other non-local storage, you must specify a local manifest path.
|
Enabling checksum reporting by editing aspera.conf
To enable checksum reporting, run the following asconfigurator command:
$ asconfigurator -x "set_node_data;file_checksum,checksum"
To enable and configure the file manifest where checksum report data is stored, run the following commands:
$ asconfigurator -x "set_node_data;file_manifest,text"
$ asconfigurator -x "set_node_data;file_manifest_path,filepath"
These commands create lines in aspera.conf as shown in the following example, where checksum type is md5, file manifest is enabled, and the path is /tmp.
<file_system>
...
<file_checksum>md5</file_checksum>
<file_manifest>text</file_manifest>
<file_manifest_path>/tmp
</file_manifest_path>
...
</file_system>
Enabling checksum reporting in an ascp session
To enable checksum reporting on a per-transfer-session basis, run ascp with the --file-checksum=hash option, where hash is sha1, md5, sha-512, sha-384, sha-256, or none (the default).
Enable the manifest with
--file-manifest=output where
output is either text
or
none
. Set the path to the manifest file with
--file-manifest-path=path.
For example:
$ ascp --file-checksum=md5 --file-manifest=text --file-manifest-path=/tmp
file aspera_user_1@189.0.202.39:/destination_path
Setting up a Pre/Post-processing Script
An alternative to enabling and configuring the file manifest to collect checksum reporting is to set up a pre/post-processing script to report the values.
The checksum of a transferred file is stored in the pre/post environment variable
FILE_CSUM
, which can be used in pre/post scripts to output file
checksums. For example, the following script outputs the checksum to the file
/tmp/cksum.log:
#!/bin/bash
if [ $TYPE == File ]; then
if [ $STARTSTOP == Stop ]; then
echo "The file is: $FILE" >> /tmp/cksum.log
echo "The file checksum is: $FILE_CSUM" >> /tmp/cksum.log
chmod 777 $FILE
fi
fi
For information on pre- and post-processing scripts and environment variables, see IBM Aspera Enterprise Server Admin Guide: Testing and Optimizing Transfer Performance.
Comparing Checksums
If you open a file that you downloaded with Aspera and find that it is corrupted, you can determine when the corruption occurred by comparing the checksum that is reported by Aspera to the checksums of the files on the destination and on the source.
- Retrieve the checksum that was calculated by Aspera as the file was
transferred.
- If you specified a file manifest and file manifest path as part of an ascp transfer or pre/post processing script, the checksums are in that file in the specified location.
- If you specified a file manifest and file manifest path in the GUI or aspera.conf, the checksums are in a file that is named aspera-transfer-transfer_id-manifest.txt in the specified location.
- Calculate the checksum of the corrupted file. This example uses the MD5 checksum method;
replace MD5 with the appropriate checksum method if you use a different
one.
$ md5 filepath
- Compare the checksum reported by Aspera with the checksum that you calculated
for the corrupted file.
- If they do not match, then corruption occurred as the file was written to the destination. Download the file again and confirm that it is not corrupted. If it is corrupted, compare the checksums again. If they do not match, investigate the write process or attempt another download. If they match, continue to the next step.
- If they match, then corruption might have occurred as the file was read from the source. Continue to the next step.
- Calculate the checksums for the file on the source. These examples use the MD5
checksum method; replace MD5 with the appropriate checksum method if you use a
different
one.Windows:
> CertUtil -hashfile filepath MD5
Mac OS X:$ md5 filepath
Linux and Linux on z Systems:# md5sum filepath
AIX:# csum -h MD5 filepath
Solaris:# digest -a md5 -v filepath
- Compare the checksum of the file on the source with the one reported by Aspera.
- If they do not match, then corruption occurred when the file was read from the source. Download the file again and confirm that it is not corrupted on the destination. If it is corrupted, continue to the next step.
- If they match, confirm that the source file is not corrupted. If the source file is corrupted, replace it with an uncorrupted one, if possible, and then download the file again.