Transferring and Deleting Files with the Aspera Watch Service

When used in conjunction with ascp commands, the Aspera Watch Service (asperawatchd) allows for fast detection and sending of new and deleted items. By comparing snapshots of the file directory it is watching, asperawatchd generates file lists for ascp commands for transfers requests. To generate snapshots and file lists, you must first configure and start asperawatchd. For more information, see Configuring the Aspera Watch Service.

Process Overview

  1. Determine the domain under which the watchd service is running.
  2. Retrieve the uuid of the watcher for the directory configured in aspera.conf.
  3. Take a snapshot of the current state of the directory.
  4. Generate a snapshot differential between the previous state and the current state and save it in a file list.
  5. Use the file list in an ascp command (to send created and modified files) or in an asdelete command (to delete files).

Viewing Active watchd Domains

To view domains under which watchd service is running, run the following command:

# /opt/aspera/bin/aswatchadmin query-daemons

The output will be similar to the following example in which root is the domain:

Found 2 daemons: root

Retrieving the Watcher uuid

To generate a file list for all the changes in a directory, first retrieve the uuid of the watcher for the directory you configured in the aspera.conf file under the <watches> section.

# /opt/aspera/bin/aswatchadmin query-watches domain
Note: If the datastore is different from the default redis:localhost:31415, specify the datastore using the argument --db-spec=datastore:host:port.

The output will have the following format:

Found a single watch: uuid, path=filepath
The uuid looks like e565a3e3-e25e-44c2-9d68-db1d59cef4c7. Copy the uuid to input into the snapshot command.

Taking a Snapshot

A snapshot represents the structure of the file system at a particular point of time; asperawatchd compares two snapshots to detect file system changes and generate a file list of those changes. To take a snapshot, run the following command with the watcher uuid associated with the file system.
# /opt/aspera/bin/aswatchadmin create-snapshot domain uuid
For example, to take a snapshot of the watch with the uuid from the previous section:
# /opt/aspera/bin/aswatchadmin create-snapshot root e565a3e3-e25e-44c2-9d68-db1d59cef4c7
Successfully created snapshot 1.

Sending New and Modified Files

Take another snapshot of the directory, generate and save a list of the files that have been created or modified since the previous snapshot, then run an ascp command to send those files to another location.
# /opt/aspera/bin/aswatchadmin snapshot-differential domain uuid snapshot_number_1 snapshot_number_2 --exclude-remove --escape + --format +PATH
# ascp --file-list /filelist --source-prefix=/prefix --mode=send --user=username --host=host /target_directory 

For example:

# /opt/aspera/bin/aswatchadmin snapshot-differential root e565a3e3-e25e-44c2-9d68-db1d59cef4c7 1 2 --exclude-remove --escape + --format +PATH
# ascp --file-list ~/filelist --source-prefix=/data/D1 --mode=send --user=aspera --host= /data/R2 
file1                                       100%   10MB    9.7Mb/s     00:07    
file2                                       100%   10MB    9.7Mb/s     00:16    
file3                                       100%   10MB    9.7Mb/s     00:26    
file4                                       100%   10MB    9.7Mb/s     00:33    
file5                                       100%   10MB    9.7Mb/s     00:42
file6                                       100%   10MB    9.7Mb/s     00:51
Completed: 61440K bytes transferred in 52 seconds
 (96454 bits/sec), in 6 files.
Tip: Aspera recommends using the source prefix option in conjunction with file lists to keep the file smaller. Alternatively, you can format the generated file list with a source prefix by specifying the prefix in the --format option. For example, specifying --format /data/D1{\$PATH} adds the prefix "/data/D1" to every entry in the file list.

Removed Files

The asdelete utility compares the source directory with the target directory and deletes extraneous files from the target directory. Use asdelete utility to remove files from a target directory.
# /opt/aspera/bin/asdelete --host host --auth-name username --auth-pass password /source_directory /target_directory
For example:
# /opt/aspera/bin/asdelete --host --auth-name root --auth-pass !XF345lui@0 /data/D1 /data/R1
Check the target directory to confirm deletion of the correct files.

Other aswatchadmin Commands

To view a list of all aswatchadmin commands, run the following:

# /opt/aspera/bin/aswatchadmin --help