Configuring the Watch Folder JSON Configuration File

To create a Watch Folder, you must provide a valid JSON configuration file, regardless of whether you are using the aswatchfolderadmin utility or the Node API. This reference provides the following information:

Requirements for a minimal JSON configuration and a sample configuration
A full JSON template
Details on the configuration optoins found in the template

Minimal JSON Configuration

A minimal JSON configuration for a watch folder uses the fields in the following table:

Field	Description
id (optional)	A unique ID for the watchfolder. If this field is not configured, a uuid is automatically generated for the watchfolder.
source_dir	The source directory, relative to the user's docroot. When `asperawatchd` detects a new file in this location, `asperawatchfolderd` starts an ascp session to transfer the file to target directory. This directory must be within the docroot set for the user running asperawatchd.
target_dir	The target directory, relative to the user's docroot. If this is a remote directory, define the remote machine in the host field and provide SSH crednetials for the remote machine in the user and pass fields.
host	The host of the target directory. The host can be specified with an IPv4 or IPv6 address. The preferred format for IPv6 addresses is x:x:x:x:x:x:x:x, where each of the eight x is a hexadecimal number of up to 4 hex digits. Zone IDs (for example, %eth0) can be appended to the IPv6 address.
user	The username for the host of the target directory.
pass	The password for the host of the target directory.

{
    "source_dir": "source_directory",
    "target_dir": "target_directory",
    "transport": {
        "host": "host",
        "user": "username",
        "pass": "password"
    }
}

For example:

{
    "source_dir": "/src",
    "target_dir": "/dest",
    "transport": {
        "host": "198.51.100.22",
        "user": "watchfolder_admin",
        "pass": "XF324cd28"
    }
}

Sample JSON Configuration File

{
   "source_dir":"/src",
   "target_dir":"/dest",
   "id":"b394d0ee-1cda-4f0d-b785-efdc6496c585",
   "cool_off":"30s",
   "snapshot_creation_period":"10s",
   "meta":{  
      "version":0,
      "name":"aspera_watchfolder"
   },
   "drop":{  
      "detection_strategy":"COOL_OFF_ONLY",
      "cool_off":"5s"
   },
   "post_processing":{  
      "local":{  
         "type":"TRANSFER_NONE",
         "archive_dir":"/watchfolder_sessions/{$UUID}_{$DATETIME}"
      }
   },
   "filters":[
      {
         "type":"GLOB",
         "pattern":"*.txt",
         "rule":"EXCLUDE"
      },
      {
         "type":"REGEX",
         "pattern":".*\\.txt",
         "rule":"INCLUDE"
      }
   ],
   "packages":{
      "timeout":"10s",
      "parsers":[
         {
            "final_transfer":"LIST",
            "filters":[
               {
                  "type":"REGEX",
                  "rule":"INCLUDE",
                  "pattern":".*\\.txt"
               }
            ]
         }
      ]
   },
   "transport":{  
      "host":"",
      "user":"aspx2",
      "pass":"",
      "proxy":"dnat://aspx1:@localhost:9001",
      "keypath":"",
      "fingerprint":"",
      "cookie":"",
      "tags":{  
      },
      "error_handling":{  
         "file":{  
            "max_retries":3,
            "retry_timeout":"3s"
         },
         "drop":{  
            "retry_period":"1m"
         }
      },
      "regular":{  
         "max_parallel":10,
         "connect_timeout":"10s",
         "policy":"FAIR",
         "min_rate":"0B",
         "target_rate":"10M",
         "tcp_port":22,
         "udp_port":33001,
         "read_blk_size":"",
         "write_blk_size":"",
         "datagram_size":"",
         "rexmsg_size":"",
         "cipher":"AES128",
         "overwrite":"DIFF",
         "resume":"NONE",
         "preserve_uid":false,
         "preserve_gid":false,
         "preserve_time":false,
         "preserve_creation_time":false,
         "preserve_modification_time":false,
         "preserve_access_time":false,
         "queue_threshold":"5s",
         "sample_period":"2s"
      },
      "growing_file":{  
         "max_parallel":8,
         "policy":"FAIR",
         "min_rate":"",
         "target_rate":"10M",
         "tcp_port":22,
         "udp_port":33001,
         "datagram_size":"",
         "cipher":"AES128",
         "completion_timeout":"5s",
         "memory":"2M",
         "chunk_size":"128K",
         "force_send_after":"2s",
         "filters":[  
            {
              "type":"REGEX",
              "rule":"INCLUDE",
              "pattern":".*\\.growing"
            }
         ]
      }
   }
}

Top Level Configuration

{
   "source_dir":"/src",
   "target_dir":"/dest",
   "id":"b394d0ee-1cda-4f0d-b785-efdc6496c585",
   "cool_off":"30s",
   "snapshot_creation_period":"10s",
   ...
}

Field	Description	Default
source_dir	The source directory, relative to the user's docroot. When `asperawatchd` detects a new file in this location, `asperawatchfolderd` starts an ascp session to transfer the file to target directory. This directory must be within the docroot set for the user running asperawatchd.	None: must be explicitly configured
target_dir	The target directory, relative to the user's docroot. If this is a remote directory, define the remote machine in the host field and provide SSH crednetials for the remote machine in the user and pass fields.	None: must be explicitly configured
id	Value used to identify a watch folder. If this field is not configured at creation, a uuid is automatically generated for and assigned to the watch folder.	None: automatically generated
cool_off	The duration watchfolder waits to take a directory snapshot to check for new files.	3s
snapshot_creation_period	The duration watchfolder uses to determine what files are included in the current drop.	3s

Meta Fields

{
   ...
   "meta":{  
      "version":0,
      "name":"aspera_watchfolder"
   },
   ...
}

Field	Description	Default
version	Specifies the current version of the configuration. When updating the configuration, this value must match the version stored by the server. Otherwise, the update is rejected.	0
name	The value specified in this field is added to the cookie reported by ascp.	N/A

Drop Fields

A watchfolder groups new or updated files it detects in its source folder into "drops". A drop is defined by the duration set by the snapshot_creation_period. All files in a given drop are transferred in the same transfer session, post-processed together, and reported as a unit.

{
   ...
   "drop":{  
      "detection_strategy":"COOL_OFF_ONLY",
      "cool_off":"5s"
   },
   ...
}

Field	Description	Default
detection_strategy	The strategy this watchfolder uses to detect files dropped into the source folder. `COOL_OFF_ONLY`: Create a drop that includes new files added within the duration of the `cool_off` field. `TOP_LEVEL_FILES`: Create a drop for each file placed in the top level of the source folder. `TOP_LEVEL_DIRS`: Create a drop for each directory placed in the top level of the source folder. This drop also includes the sub-directories and files in the top level directory.	`COOL_OFF_ONLY`
cool_off	The duration allowed for new files to be included in a drop. This field is only relevant for the `COOL_OFF_ONLY` detection strategy. Aspera recommends choosing a multiple of the specified `snapshot_creation_period` for predictable results.	5s

Post Processing Fields

{
   ...
   "post_processing":{  
      "local":{  
         "type":"TRANSFER_NONE",
         "archive_dir":"/watchfolder_sessions/{$UUID}_{$DATETIME}"
      }
   },
   ...
}

Field	Description	Default
type	Determines whether files are archived, deleted, or kept after transfer of a drop. `TRANSFER_NONE`: Files stay in the source directory. `TRANSFER_ARCHIVE`: Files in the source directory are moved to a final archive after successful transfer. `TRANSFER_DELETE`: Files in the source directory are deleted after successful transfer once the session completes. `FILE_TRANSFER_DELETE`: Files in the source directory are deleted after each successfully transfers, rather than waiting for the session to complete.	`TRANSFER_NONE`
archive_dir	The destination of archived files, relative to the user's docroot, for use when the post processing type is `TRANSFER_ARCHIVE`. The path can be determined using the following variables: `{$TIMESTAMP}` (Drop creation time in seconds since epoch) `{$DAY_OF_MONTH}` (Time format for drop's creation time) `{$MONTH}` `{$YEAR}` `{$HOUR}` `{$MINUTE}` `{$SECOND}` `{$DATETIME}` (alias for `{$YEAR}{$MONTH}{$DAY_OF_MONTH}-{$HOUR}{$MINUTE}{$SECOND}`) `{$UUID}` `{$NAME}` `{$STATE}` `{$FILE::STATE}` (such as `SUCCEEDED`, `FAILED`)	N/A

Filter Fields

{
   ...
   "filters":[
      {
         "type":"GLOB",
         "pattern":"*.txt",
         "rule":"EXCLUDE"
      },
      {
         "type":"REGEX",
         "pattern":".*\\.txt",
         "rule":"INCLUDE"
      }
   ],
   ...
}

Field	Description	Default
type	The type of filter. Supported filters are `GLOB` and `REGEX`.	N/A
pattern	The filter pattern.	N/A
rule	The rule for the filter. Supported rules are `INCLUDE` and `EXCLUDE`.	N/A

Packages Fields

{
   ...
   "packages":{
      "timeout":"10s",
      "parsers":[
         {
            "final_transfer":"LIST",
            "filters":[
               {
                  "type":"REGEX",
                  "rule":"INCLUDE",
                  "pattern":".*\\.txt"
               }
            ],
            ...
         }
      ]
   },
   ...
}

Field	Description	Default
final_transfer	Defines which file has to be transferred last. `LIST`: The last file in the package list is transferred last. `LAST_FILE_IN_LIST`: The files are transferred without any specific order.	`LIST`
filters	A list of filters as described in the Filter Fields section above.	N/A

The `transport` object

}
   ...
   "transport":{  
      "host":"198.51.100.22",
      "user":"aspx2",
      "pass":"XF324cd28",
      "token":"fiewle535etn23TEIW234n5sEWTnseonts",
      "proxy":"dnat://aspx1:XF324cd28@localhost:9001",
      "keypath":"~/.ssh/id_rsa",
      "fingerprint":"stringalsdjkfad",
   ...
   }
}

Option	Description	Default
host	The hostname of the destination node.	N/A
user	The SSH account login name.	N/A
pass	The SSH account password.	N/A
token	The token key string. Not valid for use with growing files.	N/A
proxy	The address of an Aspera high-speed proxy server. The proxy syntax is: `dnat(s)://user@server:port`	N/A
keypath	The path to the private key file used for public key authentication. Note: If a relative path is provided, the file at the relative path is checked for existence. If the relative path is not found, $HOME/.ssh/ is prepended to the relative path.	N/A
fingerprint	Fingerprint to check against server SSH host key fingerprint	N/A

Error Handling Fields

}
   ...
   "transport":{  
      ...
      "error_handling":{  
         "file":{  
            "max_retries":3,
            "retry_timeout":"3s"
         },
         "drop":{  
            "retry_period":"1m"
         }
      },
   ...
   }
}

Watch folder error handling distinguishes between two different error categories:

File-Specific Errors: These errors increase the file retry count every time a failure occurs. When the max_retries count is reached, the file is marked as failed and the session attempts to transfer the next file in the drop queue. All errors except the following:

License error
Authentication error
Any other error in establishing an ascp session

Other Errors: These errors do not increase the file retry count. If a given error re-occurs again and again, the same file is retried until the drop’s retry_period is exceeded. Then, the drop is marked as failed.

Option	Description	Default
max_retries	When the maximum retry count is reached, the file is marked as failed.	3
retry_timeout	Retry attempts are spaced by the timeout parameter.	3s
retry_period	If no bytes are transferred during this period and no file is completed, the drop and all remaining incomplete files of the drop are marked as failed.	1m

The `regular` object (for ascp Sessions)

{
   ...
   "transport":{
      ...
      "regular":{  
         "max_parallel":10,
         "connect_timeout":"10s",
         "policy":"FAIR",
         "min_rate":"0B",
         "target_rate":"10M",
         "tcp_port":22,
         "udp_port":33001,
         "read_blk_size":"",
         "write_blk_size":"",
         "datagram_size":"",
         "rexmsg_size":"",
         "cipher":"AES128",
         "overwrite":"DIFF",
         "resume":"NONE",
         "preserve_uid":false,
         "preserve_gid":false,
         "preserve_time":false,
         "preserve_creation_time":false,
         "preserve_modification_time":false,
         "preserve_access_time":false,
         "queue_threshold":"5s",
         "sample_period":"2s",
         "raw_options":["-L","/tmp/log"],
         "content_protect_password":"ear_password"
      },
      ...  
   }
}

Field	Description	Default
max_parallel	The maximum number of concurrent ascp sessions the watch folder can initiate.	10
connect_timeout	Timeout duration in seconds before considering ascp has failed.	10s
policy	Defines how ascp manages the bandwidth. The policy can be set to the following values: `FIXED` `FAIR` `HIGH` `LOW`	`FAIR`
min_rate	Minimum rate in bytes that ascp can use.	0B
target_rate	The target rate in megabytes that ascp should use.	10M
tcp_port	The TCP port used for the FASP session.	22
udp_port	The UPD port used by FASP for data transfer.	33001
read_blk_size	The read block size.	Default determined by settings in aspera.conf
write_blk_size	The write block size.	Default determined by settings in aspera.conf
datagram_size	The datagram size (MTU) for FASP.	Uses the detected path MTU.
rexmsg_size	The maximum size of a retransmission request.	Determined by ascp
cipher	The encryption cipher for file data. You can chooose `AES128` or `NONE`.	`AES128`
overwrite	Determines if a file is overwritten if it already already exists at the destination. You can choose: `NEVER` `ALWAYS` `DIFF` `OLDER` `DIFF+OLDER`	`DIFF`
resume	Determine if partial transfers are resumed. `NONE`: Always transfer the entire file `FILE_ATTRIBUTES`: Resume if file attributes match. `SPARSE_CHECKSUM`: Resume if file attributes and sparse checksum match. `FULL_CHECKSUM`: Resume if file attributes and full checksum match.	`NONE`
preserve_uid	Preserve the file owner user ID.	`false`
preserve_gid	Preserve the file owner group ID.	`false`
preserve_time	This option is equivalent to configuring `preserve_creation_time`, `preserve_modification_time`, and `preserve_access_time`.	`false`
preserve_creation_time	Set creation time of the destination be set to that of the source. If the destination is a non-Windows host, this option is ignored.	`false`
preserve_modification_time	Set the modification time of the destination file be set to that of the source.	`false`
preserve_access_time	Set the access time of the destination to be set to that of the source. The destination file ends up with the access time of the source file prior to the transfer.	`false`
queue_threshold	Watch folder controls the amount of data pushed to ascp for transferring. When the capacity is reached, the watch folder waits before pushing new data. This capacity is based on the effective bandwidth reported by ascp.	5s
sample_period	Period used to compute the current bandwidth. Used with `queue_threshold` to compute the amount of data pushed to ascp.	2s
raw_options	Enable the use of new ascp options that are not yet available in the configuration. Optionally set logging of raw options. Note: This option requires that raw options are also enabled in aspera.conf.	disabled
content_protect_password	Provide the encryption-at-rest password.	N/A

The `growing` Object (for faspstream Sessions)

{
   ...
   "transport":{
      ...
      "growing_file":{  
         "max_parallel":8,
         "policy":"FAIR",
         "min_rate":"",
         "target_rate":"10M",
         "tcp_port":22,
         "udp_port":33001,
         "datagram_size":"",
         "cipher":"AES128",
         "completion_timeout":"5s",
         "memory":"2M",
         "chunk_size":"128K",
         "force_send_after":"2s",
         "filters":[  
            {
              "type":"REGEX",
              "rule":"INCLUDE",
              "pattern":".*\\.growing"
            }
         ]
      }
   }
}

Option	Description	Default
max_parallel	The maximum number of concurrent faspstream sessions the watch folder can initiate.	8
policy	Defines how faspstream manages the bandwidth. The policy can be set to the following values: `FIXED` `FAIR` `HIGH` `LOW`	`FAIR`
min_rate	Minimum rate in bytes that faspstream can use.	0B
target_rate	The target rate in megabytes that faspstream should use.	10M
tcp_port	The TCP port used for the session.	22
udp_port	The UPD port used for data transfer.	33001
datagram_size	The datagram size (MTU) for FASP.	Uses the detected path MTU.
cipher	The encryption cipher for file data. You can chooose `AES128` or `NONE`.	`AES128`
completion_timeout	The duration before the session is considered complete. A growing file is considered completed when there is no new data for the duration of this timeout.	20s
force_send_after	Force faspstream to send data after the given time, even if the chunk is not full.	2s
memory	The maximum amount of memory the faspstream binary is allowed to use.	2M
chunk_size	Packet size for transfers over the network.	128K
filters	A file is a growing file if the filename matches the given filter.	N/A