Transferring and Deleting Files with the Aspera Watch Service

When used in conjunction with ascp commands, the Aspera Watch Service (asperawatchd) allows for fast detection and sending of new and deleted items. By comparing snapshots of the file directory it is watching, asperawatchd generates file lists for ascp transfers.

1. Create a subscription and decide how to manage its expiration.

   # /opt/aspera/bin/aswatchadmin subscribe daemon filepath [options]

   By default, subscriptions expire in 24 hours. If your snapshot comparisons will be spaced more than 24 hours apart, either set the expiration time to a duration longer than the time between snapshots (add --expire_in=seconds to the command) or send a resubscribe command periodically to maintain the subscription.

   For more information on creating subscriptions and resubscribing to them, see Managing Watch Subscriptions.

   In the following example, user aspera subscribes to /projectA/source and the subscription expires in 48 hours:

   # /opt/aspera/bin/aswatchadmin subscribe aspera /projectA/source --expire_in=172800
   [aswatchadmin subscribe] Successfully created subscription {"identifier":"bec581b3-3c34-47d7-a719-93f26f8272d1","path":"file:////projectA/source","scan_period":{"sec":9223372036854775807,"usec":999999},"expiration":"2018-03-15T07:39:21Z"}

   Record the subscription ID (the value of "identifier" in the output) for use in creating the snapshot. You can also retrieve the subscription ID later.

2. Create a snapshot.

   # /opt/aspera/bin/aswatchadmin create-snapshot daemon subscription_id

   If you do not have the subscription ID, run the following command:

   # /opt/aspera/bin/aswatchadmin query-subscriptions daemon

   In the following example, user aspera creates a snapshot of the directory that is watched by subscription bec581b3-3c34-47d7-a719-93f26f8272d1:

   # /opt/aspera/bin/aswatchadmin create-snapshot aspera bec581b3-3c34-47d7-a719-93f26f8272d1
   [aswatchadmin create-snapshot] Successfully created snapshot 1.

3. After the desired interval, create another snapshot to compare with the previous snapshot.
   The snapshot ID is automatically incremented with each create-snapshot command. For example, running the same command as the previous step outputs a new snapshot:

   # /opt/aspera/bin/aswatchadmin create-snapshot aspera bec581b3-3c34-47d7-a719-93f26f8272d1
   [aswatchadmin create-snapshot] Successfully created snapshot 2.

4. Generate the snapshot differential between the most recent snapshot and the snapshot before it.
   To create a snapshot differential that outputs a list that can be used by ascp, run the following command:

   # /opt/aspera/bin/aswatchadmin snapshot-differential daemon subscription_id snapshot_id --format=PATH 

   Where the snapshot ID is the latest snapshot. For example:

   # /opt/aspera/bin/aswatchadmin snapshot-differential aspera bec581b3-3c34-47d7-a719-93f26f8272d1 2
   /new_file.png
   /new_file.pdf

   Save the file list for use in the transfer session.

5. Send the new and modified files with ascp or ascp4.
   Use the --source-prefix option to append the watch directory path to the filepaths in the list:

   # ascp --file-list=filelist_pathname --source-prefix=prefix --mode=send --user=username --host=host target_directory

   For example:

   # ascp --file-list=/Users/aspera/filelist.txt --source-prefix=/projectA/source --mode=send --user=aspera --host=10.0.0.1 /projectA/destination 
   new_file.png       100%   10MB    9.7Mb/s     00:07    
   new_file.pdf       100%   100MB   9.7Mb/s     00:35    
   Completed: 112640K bytes transferred in 42 seconds
    (268190 bits/sec), in 2 files.

# /opt/aspera/bin/asdelete --host host --auth-name username --auth-pass password /source_directory /target_directory

# /opt/aspera/bin/asdelete --host 10.0.0.1 --auth-name aspera --auth-pass !XF345lui@0 /projectA/source /projectA/destination