Transferring and Deleting Files with the Aspera Watch Service

When used in conjunction with ascp commands, the Aspera Watch Service (asperawatchd) allows for fast detection and sending of new and deleted items. By comparing snapshots of the file directory it is watching, asperawatchd generates file lists for ascp commands for transfers requests. To generate snapshots and file lists, you must first configure and start asperawatchd. For more information, see Configuring the Aspera Watch Service.

Process Overview

1. Retrieve the uuid of the watcher for the directory configured in aspera.conf.
2. Take a snapshot of the current state of the directory.
3. Generate a snapshot differential between the previous state and the current state and save it in a file list.
4. Use the file list in an ascp command (to send created and modified files) or in an asdelete command (to delete files).

Retrieving the Watcher uuid

To generate a file list for all the changes in a directory, first retrieve the uuid of the watcher for the directory you configured in the aspera.conf file under the <watches> section.

# /opt/aspera/bin/aswatchadmin --db-spec=datastore:host:port --tool daemon --get-by-path=~/

For example, if the directory set in the <watches> section is /data/D1, run this command:

# /opt/aspera/bin/aswatchadmin --db-spec=redis:localhost:31415 --tool daemon --get-by-path=/data/D1
[aswatchadmin::daemon] Successfully retrieved watcher 
[c=[uuid=e565a3e3-e25e-44c2-9d68-db1d59cef4c7,p=/data/D1,i_c=true,f=[],
in=[t=CHANNEL,n='wt:e565a3e3-e25e-44c2-9d68-db1d59cef4c7:input'],
in_b=[t=CHANNEL,n='wt:e565a3e3-e25e-44c2-9d68-db1d59cef4c7:broadcast'],
t=[c=[ss.max=10000],p=[p=[t=REDIS,db=wt:e565a3e3-e25e-44c2-9d68-db1d59cef4c7@localhost:31415],c=1]]]].

In the results, find the string for the uuid (for example, uuid=e565a3e3-e25e-44c2-9d68-db1d59cef4c7) and copy the uuid.

Taking a Snapshot

A snapshot represents the structure of the file system at a particular point of time; asperawatchd compares two snapshots to detect file system changes and generate a file list of those changes. To take a snapshot, run the following command with the watcher uuid associated with the file system.

# /opt/aspera/bin/aswatchadmin --db-spec=redis:localhost:31415 --tool=instance --uuid=watch_uuid --snapshot

For example, to take a snapshot with the watcher with the uuid we retrieved in the previous section:

# /opt/aspera/bin/aswatchadmin --db-spec=redis:localhost:31415 --tool=instance --uuid=e565a3e3-e25e-44c2-9d68-db1d59cef4c7 --snapshot
[aswatchadmin::instance] Successfully created snapshot 1.

Sending New and Modified Files

Take a snapshot of the directory. Generate and save a file list of all files that have been created or modified. Then, run an ascp command with that file list to send those files to another location.

# /opt/aspera/bin/aswatchadmin--db-spec datastore:host:port --tool tree --uuid=uuid_no --diff=snapshot_version:snapshot_version --format {\$PATH} --exclude-remove > /filelist
# ascp --file-list /filelist --source-prefix=/prefix --mode=send --user=username --host=host /target_directory\target_directory 

For example:

# /opt/aspera/bin/aswatchadmin --db-spec redis:localhost:31415 --tool tree --uuid=e565a3e3-e25e-44c2-9d68-db1d59cef4c7 --diff=0:1 --format {\$PATH} --exclude-remove > ~/filelist 
# ascp --file-list ~/filelist --source-prefix=/data/D1 --mode=send --user=aspera --host=10.0.0.1 /data/R2 
file1                                       100%   10MB    9.7Mb/s     00:07    
file2                                       100%   10MB    9.7Mb/s     00:16    
file3                                       100%   10MB    9.7Mb/s     00:26    
file4                                       100%   10MB    9.7Mb/s     00:33    
file5                                       100%   10MB    9.7Mb/s     00:42
file6                                       100%   10MB    9.7Mb/s     00:51
Completed: 61440K bytes transferred in 52 seconds
 (96454 bits/sec), in 6 files.

Removed Files

The asdelete utility compares the source directory with the target directory and deletes extraneous files from the target directory. Use asdelete utility to remove files from a target directory.

# /opt/aspera/bin/asdelete --host host --auth-name username --auth-pass password /source_directory /target_directory

For example:

# /opt/aspera/bin/asdelete --host 10.0.0.1 --auth-name root --auth-pass !XF345lui@0 /data/D1 /data/R1

Check the target directory to confirm deletion of the correct files.