Reporting Checksums

File checksums are useful for trouble-shooting file corruption, allowing you to determine at what point in the transfer file corruption occurred. Aspera servers can report source file checksums that are calculated on-the-fly during transfer and then sent from the source to the destination. To do so, the transfer must meet both of the following requirements:

  • Both the server and client computers must be running Enterprise Server, Connect Server, or Point-to-Point Client version 3.4.2 or higher.
  • The transfer must be encrypted. Encryption is enabled by default.

The user on the destination can calculate a checksum for the received file and compare it (manually or programmatically) to the checksum reported by the sender. The checksum reported by the source can be retrieved in the destination logs, a manifest file, in Aspera Console, or as an environment variable. Instructions for comparing checksums follow the instructions for enabling checksum reporting.

Checksum reporting is disabled by default. Enable and configure checksum reporting on the server by using the following methods:

  • Edit aspera.conf with asconfigurator.
  • Set options in the client GUI.
  • Set ascp command-line options (per-transfer configuration).

Command-line options override the settings in aspera.conf and the GUI.

Important: When checksum reporting is enabled, transfers of very large files (>TB) take a long time to resume because the entire file must be reread.

Overview of Checksum Configuration Options

asconfigurator Option

ascp Option

Description

file_checksum

--file-checksum=type

Enable checksum reporting and specify the type of checksum to calculate for transferred files.
  • any - Allow the checksum format to be whichever format the client requests. (Default in aspera.conf)
  • md5 - Calculate and report an MD5 checksum.
  • sha1 - Calculate and report a SHA-1 checksum.
  • sha256 - Calculate and report a SHA-256 checksum.
  • sha384 - Calculate and report a SHA-384 checksum.
  • sha512 - Calculate and report a SHA-512 checksum.
Note: The default value for the ascp option is none, in which case the reported checksum is the one configured on the server, if any.

file_manifest

--file_manifest=output

The file manifest is a file that contains a list of content that was transferred in a transfer session. The file name of the file manifest is automatically generated from the transfer session ID.

When set to none, no file manifest is created. (Default)

When set to text, a text file is generated that lists all files in each transfer session.

file_manifest_path

--file_manifest_path=path

The location where manifest files are written. The location can be an absolute path or a path relative to the transfer user's home directory. If no path is specified (default), the file is generated under the destination path at the receiver, and under the first source path at the sender.
Note: File manifests can be stored only locally. Thus, if you are using S3 or other non-local storage, you must specify a local manifest path.

Enabling checksum reporting by editing aspera.conf

To enable checksum reporting, run the following asconfigurator command:

$ asconfigurator -x "set_node_data;file_checksum,checksum"

To enable and configure the file manifest where checksum report data is stored, run the following commands:

$ asconfigurator -x "set_node_data;file_manifest,text"
$ asconfigurator -x "set_node_data;file_manifest_path,filepath"

These commands create lines in aspera.conf as shown in the following example, where checksum type is md5, file manifest is enabled, and the path is /tmp.

<file_system>
    ...
    <file_checksum>md5</file_checksum>
    <file_manifest>text</file_manifest>
    <file_manifest_path>/tmp</file_manifest_path>
    ...
</file_system>

Enabling checksum reporting in an ascp session

To enable checksum reporting on a per-transfer-session basis, run ascp with the --file-checksum=hash option, where hash is sha1, md5, sha-512, sha-384, sha-256, or none (the default).

Enable the manifest with --file-manifest=output where output is either text or none. Set the path to the manifest file with --file-manifest-path=path.

For example:

$ ascp --file-checksum=md5 --file-manifest=text --file-manifest-path=/tmp file aspera_user_1@189.0.202.39:/destination_path

Setting up a Pre/Post-processing Script

An alternative to enabling and configuring the file manifest to collect checksum reporting is to set up a pre/post-processing script to report the values.

The checksum of a transferred file is stored in the pre/post environment variable FILE_CSUM, which can be used in pre/post scripts to output file checksums. For example, the following script outputs the checksum to the file /tmp/cksum.log:

#!/bin/bash
if [ $TYPE == File ]; then
    if [ $STARTSTOP == Stop ]; then
        echo "The file is: $FILE" >> /tmp/cksum.log
        echo "The file checksum is: $FILE_CSUM" >> /tmp/cksum.log
        chmod 777 $FILE
    fi 
fi

For information on pre- and post-processing scripts and environment variables, see IBM Aspera Enterprise Server Admin Guide: Testing and Optimizing Transfer Performance.

Comparing Checksums

If you open a file that you downloaded with Aspera and find that it is corrupted, you can determine when the corruption occurred by comparing the checksum that is reported by Aspera to the checksums of the files on the destination and on the source.

  1. Retrieve the checksum that was calculated by Aspera as the file was transferred.
    • If you specified a file manifest and file manifest path as part of an ascp transfer or pre/post processing script, the checksums are in that file in the specified location.
    • If you specified a file manifest and file manifest path in the GUI or aspera.conf, the checksums are in a file that is named aspera-transfer-transfer_id-manifest.txt in the specified location.
  2. Calculate the checksum of the corrupted file. This example uses the MD5 checksum method; replace MD5 with the appropriate checksum method if you use a different one.
    $ md5 filepath
  3. Compare the checksum reported by Aspera with the checksum that you calculated for the corrupted file.
    • If they do not match, then corruption occurred as the file was written to the destination. Download the file again and confirm that it is not corrupted. If it is corrupted, compare the checksums again. If they do not match, investigate the write process or attempt another download. If they match, continue to the next step.
    • If they match, then corruption might have occurred as the file was read from the source. Continue to the next step.
  4. Calculate the checksums for the file on the source. These examples use the MD5 checksum method; replace MD5 with the appropriate checksum method if you use a different one.
    Windows:
    > CertUtil -hashfile filepath MD5
    Mac OS X:
    $ md5 filepath
    Linux and Linux on z Systems:
    # md5sum filepath
    AIX:
    # csum -h MD5 filepath
    Solaris:
    # digest -a md5 -v filepath
  5. Compare the checksum of the file on the source with the one reported by Aspera.
    • If they do not match, then corruption occurred when the file was read from the source. Download the file again and confirm that it is not corrupted on the destination. If it is corrupted, continue to the next step.
    • If they match, confirm that the source file is not corrupted. If the source file is corrupted, replace it with an uncorrupted one, if possible, and then download the file again.