Watch Folder Configuration Reference

When you create or edit a Watch Folder in the GUI, you must specify the Watch Folder service, the source path, Watch scan period, and the remote connection. Other settings can be left with their default values or configured to meet your requirements. The following tables describe available Watch Folder settings by tab.

Watch Folder

These settings configure the Watch Folder source and destination, and the connection to the remote host.

Setting Description Default
Watch Folder Service The Watch Folder service, defined by its user and service ID, that the Watch Folder uses. The first Watch Folder service in the list of services alphabetized by username.
Watch Folder name A unique name for the Watch Folder. The value specified in this field is added to the cookie reported by ascp. Optional. None specified
Watchd scan period How frequently the Watch Service scans the source path for file system changes. For file systems with file notifications (Linux, Windows, macOS), set to 30 minutes (30m) to provide a backup to the notification system, or set to "infinite" to never scan.

On file systems without file notifications, such as object storage, mounted storage (NFS), Solaris, AIX, and Isilon, file system scans triggered by the scan period are used to detect file changes. In this case, set the scan period to frequently scan for changes. On operating systems that support file notifications (Linux, Windows, macOS), asperawatchd uses the file notifications as the primary means for detecting changes, and the scan period serves as a backup. In this case, the default value of 30 minutes is usually acceptable and no change is necessary. To never scan, and rely entirely on file notifications, set to infinite.

For pull Watch Folders, file systems scans that are triggered by scan_period are the sole means for detecting changes in the source directory.

Lower scan periods detect changes faster but can result in greater resource consumption, particularly for object storage.

30m
Direction Select Push to transfer from the local computer to a remote server.Select Pull to transfer from the remote server to the local computer. Push
Local path Click Browse to select the local path. The local path is the source for push Watch Folders, and the target for pull Watch Folders. None specified
Host The IP address, DNS, hostname, or URL of the remote server. If you enter the host manually, use the following syntax based on the type of remote endpoint and authentication method:
  • HST Server or HST Endpoint authenticated with an SSH user: Enter the IP address or hostname of the endpoint for the host, then enter the SSH user and their password or public key. (Note: This method is not supported for pull Watch Folders)
  • HST Server or HST Endpoint authenticated with Node API or access key credentials: Enter the node URL as https://ip_address_or_server_url:9092/. If a different HTTPS port is configured, replace 9092 with the correct port. Enter the Node API username or access key ID as the User and the Node API user's password or the access key secret as the Password.
    Note: Node API authentication uses HTTPS, even if HTTP is specified in the node URL.
  • IBM Aspera on Cloud (including IBM Aspera on Cloud transfer service nodes) and IBM Aspera Transfer Cluster Manager nodes: Enter the endpoint URL as https://ip_address_or_server_url:443/ and provide the access key ID and secret for the User and Password. (Note: These remote endpoint types are not supported for pull Watch Folders)
  • IBM Aspera Shares: Enter the URL of the Shares server as https://ip_address:443 and provide the Shares login credentials.
None specified
User The username for authentication. Required. Depending on the type of authentication, it is the transfer user's username, Node API username, Shares username, or access key ID. For SSH, the system account username. For a Shares server, the Shares username. For a node URL host, the Node API username or access key ID. None specified
Authentication For SSH authentication, select Password and enter the user's password, or select Public Key and select the user's public key from the drop-down menu.

For Shares hosts, enter the Shares user's password.

For node authentication, enter the Node API user's password or access key secret.

Password
SSH Port (TCP) The port to use for SSH connections. 22
FASP port (UDP) The port to use for UDP connections. 33001
SSH host key fingerprint The SSH fingerprint of the remote server. Aspera strongly recommends using SSH fingerprint for security. If the fingerprint does not match that of the server, the transfer fails with the error "Remote host is not who we expected". For more information, see Securing Your SSH Server ("Configuring Transfer Server Authentication"). None specified
Connection timeout How long to wait for a connection to respond before failing. 10s
Proxy URL If using, the address of an IBM Aspera Proxy server. The proxy syntax is: dnat(s)://user:password@server:port None specified
Remote path Click Browse to select the source or target directory on the remote server. The remote path must be within the user's docroot.

For access key authentication, the path is relative to the storage specified in the access key.

None specified

Settings

These settings define how Watch Folder watches the file system and groups new files added to the source folder into "drops". Drops are groups of files that are transferred together in one session, post-processed together, and reported as a unit.

Setting Description Default
Sample period Period used to compute the current bandwidth. Used with queue_threshold to compute the amount of data pushed to ascp. 2s
Queue threshold Watch Folders controls the amount of data pushed to ascp for transferring. When the capacity is reached, Watch Folders waits before pushing new data. This capacity is based on the effective bandwidth reported by ascp. 5s
Snapshot creation period The interval during which Watch Folders groups new files in the source directory into a drop. All files in a drop are transferred in the same transfer session, post-processed together, and reported as a unit. Watch Folders uses asperawatchd to detect file system modifications, and continuously creates snapshots to compute the snapshot differential. A small value results in high temporal resolution for detecting file system modifications, whereas a large value improves asperawatchd performance. Default: 3s. 3s
Detection strategy The strategy that Watch Folders uses to create drops when new files are added to the source folder:
  • Cool off only: The drop includes new files that are added to the source folder within the cool off period.
  • Top level files: Create a drop for each file added to the top level of the source folder.
  • Top level dirs: Create a drop for each directory added to the top level of the source folder. The drop includes new subdirectories and files in the top-level directory.
Cool off only
(Drops) Detection cool off The time after the first new file is added to the source file during which any other new files are included in the same drop. This setting is only relevant if the detection strategy is Cool off only. Aspera recommends setting the detection cool off to a multiple of the Snapshot creation period for predictable results. 5s
Maximum parallel ascp The maximum number of concurrent ascp sessions that Watch Folders can start. 10
(Files) Detection cool off How long the Watch Folder service waits for files in the watched folder to stop changing (stabilize) before taking a directory snapshot and creating a drop. Default: 5s. 3s
Filters Restrict the files that are included in the Watch Folder transfer by setting filters.

Click Files matching any of these filters are excluded from watch folder transfers to add a filter.

Setting an Include filter protects matching files from subsequent Exclude filters. Filters are applied in order. Watch Folders supports glob and Regex filters. The glob filter system is the same as Ascp; see Using Filters to Include and Exclude Files.

Note: An include rule must be followed by at least one exclude rule, otherwise all files are transferred because none are excluded. To exclude all files that do not match the include rule, use /** for glob or .* for Regex.
Click Remove this filter to delete the filter.
None specified

Transfer

These settings configure the ascp transfer sessions that transfer files in each drop.

Setting Description Default
Bandwidth policy
  • high - Adjust the transfer rate to fully utilize the available bandwidth up to the maximum rate. When congestion occurs, the transfer rate is twice as fast as a fair-policy transfer. The high policy requires maximum (target) and minimum transfer rates.
  • fair - Adjust the transfer rate to fully utilize the available bandwidth up to the maximum rate. When congestion occurs, bandwidth is shared fairly by transferring at an even rate. The fair policy requires maximum (target) and minimum transfer rates.
  • low - Adjust the transfer rate to use the available bandwidth up to the maximum rate. Similar to fair mode, but less aggressive when sharing bandwidth with other network traffic. When congestion occurs, the transfer rate is reduced to the minimum rate until other traffic decreases.
  • fixed - Attempt to transfer at the specified target rate, regardless of network or storage capacity. This can decrease transfer performance and cause problems on the target storage. Aspera discourages using the fixed policy except in specific contexts, such as bandwidth testing. The fixed policy requires a maximum (target) rate.
Fair
Target rate The target transfer rate. Transfer at rates up to the specified target rate. This option accepts suffixes T for terabits/s, G for gigabits/s, M for megabits/s, K for kilobits/s, or B for bits/s. Decimals are allowed. If this option is not set by the client, the setting in the server's aspera.conf is used. If a rate cap is set in the local or server aspera.conf, the rate does not exceed the cap. 10.00 Mbps
Minimum rate Attempt to transfer no slower than the specified minimum transfer rate. 0 bps
Transport Encryption
Select the encryption cipher. Aspera supports three sizes of AES cipher keys (128, 192, and 256 bits) and supports two encryption modes, cipher feedback mode (CFB) and Galois/counter mode (GCM). The GCM mode encrypts data faster and increases transfer speeds compared to the CFB mode, but the server must support and permit it.

Cipher rules

The encryption cipher that you are allowed to use depends on the server configuration and the version of the client and server:

  • When you request a cipher key that is shorter than the cipher key that is configured on the server, the transfer is automatically upgraded to the server configuration. For example, when the server setting is AES-192 and you request AES-128, the server enforces AES-192.
  • When the server requires GCM, you must use GCM (requires version 3.9.0 or newer) or the transfer fails.
  • When you request GCM and the server is older than 3.8.1 or explicity requires CFB, the transfer fails.
  • When the server setting is "any", you can use any encryption cipher. The only exception is when the server is 3.8.1 or older and does not support GCM mode; in this case, you cannot request GCM mode encryption.
  • When the server setting is "none", you must use "none". Transfer requests that specify an encryption cipher are refused by the server.

Cipher Values

Value Description Support

AES-128
AES-192
AES-256

Use the GCM or CFB encryption mode, depending on the server configuration and version (see cipher negotiation matrix). All client and server versions.

AES-128-CFB
AES-192-CFB
AES-256-CFB

Use the CFB encryption mode. Clients version 3.9.0 and newer, all server versions.

AES-128-GCM
AES-192-GCM
AES-256-GCM

Use the GCM encryption mode. Clients and servers version 3.9.0 and newer.
NONE Do not encrypt data in transit. Aspera strongly recommends against using this setting. All client and server versions.

Client-Server Cipher Negotiation

The following table shows which encryption mode is used depending on the server and client versions and settings:

  Server, v3.9.0+

AES-XXX-GCM

Server, v3.9.0+

AES-XXX-CFB

Server, v3.9.0+

AES-XXX

Server, v3.8.1 or older

AES-XXX

Client, v3.9.0+

AES-XXX-GCM

GCM server refuses transfer GCM server refuses transfer
Client, v3.9.0+

AES-XXX-CFB

server refuses transfer CFB CFB CFB
Client, v3.9.0+

AES-XXX

GCM CFB CFB CFB
Client, v3.8.1 or older

AES-XXX

server refuses transfer CFB CFB CFB
AES-128
SSH host key fingerprint The SSH fingerprint of the remote server. Aspera strongly recommends using SSH fingerprint for security. If the fingerprint does not match that of the server, the transfer fails with the error "Remote host is not who we expected". For more information, see Securing Your SSH Server ("Configuring Transfer Server Authentication"). None specified
Token If required, the token string. Not valid for use with growing files. None specified
Tags Specify custom metadata in JSON format. The tags object is passed directly to the ascp session. For more information on writing custom metadata for uploads to object storage (as in the example), see Writing Custom Metadata for Objects in Object Storage. None specified
Read block size The read block size. None specified (uses the value set in aspera.conf).
Write block size The write block size. None specified (uses the value set in aspera.conf).
Datagram size The datagram size (MTU) for FASP. None specified (uses the detected path MTU).
Rex message size The maximum size of a retransmission request. Maximum: 1440. None specified
Raw ascp options
Specify ascp options and their arguments that are not yet available in Watch Folders to apply to Watch Folder transfers. To use raw options, they must be enabled in the client's aspera.conf by running the following command:
# asconfigurator -x "set_central_server_data;raw_options,enable"
None specified
Cookie Each transfer session (drop) includes a cookie that provides information about the Watch Folder that initiates the transfer session and the drop. The cookie has the following format:
aspera:watchfolder:watchfolder_id:watchfolder_name:drop-id:drop_name
  • watchfolder_id: The Watch Folder ID
  • watchfolder_name: The Watch Folder name (as specified for Watch Folder name in the Watch Folder settings)
  • drop_id: The drop ID that is automatically generated by Watch Folders
  • drop_name: The drop name. Unless otherwise specified, the drop name is generated from the file name.

You can override the default values by specifying your own.

None specified (use default values)
Content protection Enter a password to enable client-side encryption at rest. Files that are uploaded to the server are appended with a .aspera-env extension. To download and decrypt .aspera-env files from the server, the client must provide the password. For more information on client-side encryption at rest, see Client-Side Encryption-at-Rest (EAR).

To disable content protection, clear the check box.

None specified (content protection disabled)
Number of retry attempts How many times to try transferring a file before the file is marked as failed. 3
Wait between retries How frequently to retry file transfers. 3s
Retry maximum for If no bytes are transferred during the specified period and no file is completed, the drop and all remaining incomplete files in the drop are marked as failed. 1m

File Handling

These settings configure how files and their attributes are handled on the source and destination.

Setting Description Default
Resume policy Specify if partial files are resumed. To always re-transfer files, select Never. To resume conditionally, select from the following options:
  • Compare file attributes - Compares the sizes of the existing and original file. If they are the same, then the transfer resumes, otherwise the original file is transferred again.
  • Compare sparse file checksums - Performs a sparse checksum on the existing file and resumes the transfer if the file matches the original, otherwise the original file is transferred again. (Default)
  • Compare full file checksums - Performs a full checksum on the existing file and resumes the transfer if the file matches the original, otherwise the original file is transferred again.

When resuming file transfer is enabled, select an overwrite rule for when the same file exists at the destination. Click the drop-down menu for If a complete file already exists at the destination.

Never
Preserve file UIDs and GIDs Select to preserve the file owner user ID or group ID. None selected
Preserve file timestamps Select to preserve all file timestamps. Equivalent to enabling Preserve creation timestamps, Preserve modification timestamps, and Preserve access timestamps.
Note: Access, modification, and source access times cannot be preserved for node and Shares connections that are using cloud storage.
Not enabled
Preserve creation timestamps Set creation time of the destination be set to that of the source. If the destination is a non-Windows host, this option is ignored. Not enabled
Preserve modification timestamps Set the modification time of the destination file to that of the source. Not enabled
Preserve access timestamps Set the access time of the destination to that of the source. The destination file has the access time of the source file prior to the transfer. Not enabled
Source Handling Select what to do with source files after they are successfully transferred to the destination. Files can be archived, deleted, or retained after transfer of a drop. When files are archived or deleted, source sub-directories are also deleted from the source, unless the sub-directories were empty to start. File structure is preserved in the archive.

File archiving (Automatically move source files to an archive folder) is not supported for sources in object storage.

  • Do nothing: Source files remain in the Watch Folder after transfer.
  • Automatically move source files to an archive folder: Move source files to the specified archive folder. The archive path can use the following variables:
    • {$TIMESTAMP} (Drop creation time in seconds since epoch)
    • {$DAY_OF_MONTH} (Time format for drop's creation time)
    • {$MONTH}
    • {$YEAR}
    • {$HOUR}
    • {$MINUTE}
    • {$SECOND}
    • {$DATETIME} (alias for {$YEAR}{$MONTH}{$DAY_OF_MONTH}-{$HOUR}{$MINUTE}{$SECOND})
    • {$UUID}
    • {$NAME}
    • {$STATE}
    • {$FILE::STATE} (such as SUCCEEDED, FAILED)
  • Automatically delete source files (once drop is done): Delete source files after the entire transfer session is complete and successful.
  • Automatically delete source file (immediately after transfer): Delete source files as they are successfully transferred, rather than waiting for the entire session to complete before deleting files.
Do nothing after transfer

Growing Files

Growing files (files that are written to the source folder from a streaming input) are transferred using FASPStream technology rather than ascp. Identify growing files by specifying filters; files that match the filters are considered growing files. This feature requires a growing files-enabled license.

Note: Growing files are only supported for local sources (push Watch Folders) and must be authenticated by a transfer user (password or SSH key file). The transfer user cannot be restricted to aspshell and the source cannot be in object storage.

These settings configure how growing files are identified and the transfer session.

Setting Description Default
Maximum active drops The maximum number of concurrent FASPStream sessions the Watch Folder can initiate. 8
Bandwidth policy
  • high - Adjust the transfer rate to fully utilize the available bandwidth up to the maximum rate. When congestion occurs, the transfer rate is twice as fast as a fair-policy transfer. The high policy requires maximum (target) and minimum transfer rates.
  • fair - Adjust the transfer rate to fully utilize the available bandwidth up to the maximum rate. When congestion occurs, bandwidth is shared fairly by transferring at an even rate. The fair policy requires maximum (target) and minimum transfer rates.
  • low - Adjust the transfer rate to use the available bandwidth up to the maximum rate. Similar to fair mode, but less aggressive when sharing bandwidth with other network traffic. When congestion occurs, the transfer rate is reduced to the minimum rate until other traffic decreases.
  • fixed - Attempt to transfer at the specified target rate, regardless of network or storage capacity. This can decrease transfer performance and cause problems on the target storage. Aspera discourages using the fixed policy except in specific contexts, such as bandwidth testing. The fixed policy requires a maximum (target) rate.
Fair
Target rate The target transfer rate. Transfer at rates up to the specified target rate. This option accepts suffixes T for terabits/s, G for gigabits/s, M for megabits/s, K for kilobits/s, or B for bits/s. Decimals are allowed. If this option is not set by the client, the setting in the server's aspera.conf is used. If a rate cap is set in the local or server aspera.conf, the rate does not exceed the cap. 10.00 Mbps
Minimum rate Attempt to transfer no slower than the specified minimum transfer rate. 0 bps
Datagram size The datagram size (MTU) for FASP. None specified
Transport encryption Select the encryption cipher (AES-128) to use for encrypting data in transit, or disable encryption by selecting none. AES-128
SSH Port (TCP) The port to use for SSH connections. 22
FASP Port (UDP) The port to use for UDP connections. 33001
Completion timeout How long to wait before the session is considered complete. A growing file is considered complete when no new data arrives within the timeout period. 5s
Send data after maximum Force FASPStream to send data after the given time, even if the chunk is not full. 2s
Memory The maximum amount of memory FASPStream is allowed to use. 2.00 MB
Chunk size Packet size for transfers over the network. 128.00 KB
Filters Identify growing files as those that match the specified filters. Click Files matching any of these filters are excluded from Watch Folder transfers to add a filter. Set an Include filter that matches your growing files.Filters are applied in order. Watch Folders supports glob and Regex filters. The glob filter system is the same as Ascp; see Using Filters to Include and Exclude Files.
Note: An include rule must be followed by at least one exclude rule, otherwise all files are transferred because none are excluded. To exclude all files that do not match the include rule, use /** for glob or .* for Regex.

Click Remove this filter to delete a filter.

None specified

Packages

These settings enable the use of package files to define the order of files in the transfer queue for a session, and specify which file, either the last in the list or the package file itself, to transfer last. The package file is identified by setting matching filters.

Setting Description Default
Package timeout How long to wait for file dependencies to be satisfied (files that must be transferred before the last file are transferred) before considering the dependency as unsatisfied. 10s
Parsers Click Add a package parser to define the file to transfer last in the transfer session. Select:
  • List: The package file is transferred last, after all files in the package file successfully transfer.
  • Last file in list: The last file in the package file is transferred last.

Identify package files as those that match the specified filters. Click Files matching any of these filters are excluded from Watch Folder transfers to add a filter. Set an Include filter that matches your package files. Filters are applied in order. Watch Folders supports glob and Regex filters. The glob filter system is the same as Ascp; see Using Filters to Include and Exclude Files.

Note: An include rule must be followed by at least one exclude rule, otherwise all files are transferred because none are excluded. To exclude all files that do not match the include rule, use /** for glob or .* for Regex.

Click Remove this filter to delete the filter.

None specified