Configuring the Watch Folder JSON Configuration File

To create a watchfolder, you must provide a valid JSON configuration file, regardless of whether you are using the aswatchfolderadmin utility or the Node API. This reference provides the following information:

Minimal JSON Configuration

A minimal JSON configuration for a watch folder uses the fields in the following table:

Field Description
id (optional) A unique ID for the watchfolder. If this field is not configured, a uuid is automatically generated for the watchfolder.
source_dir The source directory. When asperawatchd detects a new file in this location, asperawatchfolderd starts an ascp session to transfer the file to target directory. This directory must be within the docroot set for the user running asperawatchd.
target_dir The target directory. If this is a remote directory, define the remote machine in the host field and provide SSH crednetials for the remote machine in the user and pass fields.
host The host of the target directory.
user The username for the host of the target directory.
pass The password for the host of the target directory.
{
    "source_dir": "source_directory",
    "target_dir": "target_directory",
    "transport": {
        "host": "host",
        "user": "username",
        "pass": "password"
    }
}
For example:
{
    "source_dir": "C:\src",
    "target_dir": "C:\dest",
    "transport": {
        "host": "198.51.100.22",
        "user": "watchfolder_admin",
        "pass": "XF324cd28"
    }
}

Sample JSON Configuration File

{
   "source_dir":"C:\src",
   "target_dir":"C:\dest",
   "id":"b394d0ee-1cda-4f0d-b785-efdc6496c585",
   "cool_off":"30s",
   "snapshot_creation_period":"10s",
   "meta":{  
      "version":0,
      "name":"aspera_watchfolder"
   },
   "drop":{  
      "detection_strategy":"COOL_OFF_ONLY",
      "cool_off":"5s"
   },
   "post_processing":{  
      "local":{  
         "type":"TRANSFER_NONE",
         "archive_dir":"C:\watchfolder_sessions\{$UUID}_{$DATETIME}"
      }
   },
   "filters":[
      {
         "type":"GLOB",
         "pattern":"*.txt",
         "rule":"EXCLUDE"
      },
      {
         "type":"REGEX",
         "pattern":".*\\.txt",
         "rule":"INCLUDE"
      }
   ],
   "packages":{
      "timeout":"10s",
      "parsers":[
         {
            "final_transfer":"LIST",
            "filters":[
               {
                  "type":"REGEX",
                  "rule":"INCLUDE",
                  "pattern":".*\\.txt"
               }
            ]
         }
      ]
   },
   "transport":{  
      "host":"",
      "user":"aspx2",
      "pass":"",
      "proxy":"dnat://aspx1:@localhost:9001",
      "keypath":"",
      "fingerprint":"",
      "cookie":"",
      "tags":{  
      },
      "error_handling":{  
         "file":{  
            "max_retries":3,
            "retry_timeout":"3s"
         },
         "drop":{  
            "retry_period":"1m"
         }
      },
      "regular":{  
         "max_parallel":10,
         "connect_timeout":"10s",
         "policy":"FAIR",
         "min_rate":"0B",
         "target_rate":"10M",
         "tcp_port":22,
         "udp_port":33001,
         "read_blk_size":"",
         "write_blk_size":"",
         "datagram_size":"",
         "rexmsg_size":"",
         "cipher":"AES128",
         "overwrite":"DIFF",
         "resume":"NONE",
         "preserve_uid":false,
         "preserve_gid":false,
         "preserve_time":false,
         "preserve_creation_time":false,
         "preserve_modification_time":false,
         "preserve_access_time":false,
         "queue_threshold":"5s",
         "sample_period":"2s"
      },
      "growing_file":{  
         "max_parallel":8,
         "policy":"FAIR",
         "min_rate":"",
         "target_rate":"10M",
         "tcp_port":22,
         "udp_port":33001,
         "datagram_size":"",
         "cipher":"AES128",
         "completion_timeout":"5s",
         "memory":"2M",
         "chunk_size":"128K",
         "force_send_after":"2s",
         "filters":[  
            {
              "type":"REGEX",
              "rule":"INCLUDE",
              "pattern":".*\\.growing"
            }
         ]
      }
   }
}

Top Level Configuration

{
   "source_dir":"C:\src",
   "target_dir":"C:\dest",
   "id":"b394d0ee-1cda-4f0d-b785-efdc6496c585",
   "cool_off":"30s",
   "snapshot_creation_period":"10s",
   ...
}
Field Description Default
source_dir The source directory. When asperawatchd detects a new file in this location, asperawatchfolderd starts an ascp session to transfer the file to target directory. This directory must be within the docroot set for the user running asperawatchd. None: must be explicitly configured
target_dir The target directory. If this is a remote directory, define the remote machine in the host field and provide SSH crednetials for the remote machine in the user and pass fields. None: must be explicitly configured
id Value used to identify a watch folder. If this field is not configured at creation, a uuid is automatically generated for and assigned to the watch folder. None: automatically generated
cool_off The duration watchfolder waits to take a directory snapshot to check for new files. 3s
snapshot_creation_period The duration watchfolder uses to determine what files are included in the current drop. 3s

Meta Fields

{
   ...
   "meta":{  
      "version":0,
      "name":"aspera_watchfolder"
   },
   ...
}
Field Description Default
version Specifies the current version of the configuration. When updating the configuration, this value must match the version stored by the server. Otherwise, the update is rejected. 0
name The value specified in this field is added to the cookie reported by ascp. N/A

Drop Fields

A watchfolder groups new or updated files it detects in its source folder into "drops". A drop is defined by the duration set by the snapshot_creation_period. All files in a given drop are transferred in the same transfer session, post-processed together, and reported as a unit.

{
   ...
   "drop":{  
      "detection_strategy":"COOL_OFF_ONLY",
      "cool_off":"5s"
   },
   ...
}
Field Description Default
detection_strategy The strategy this watchfolder uses to detect files dropped into the source folder.
  • COOL_OFF_ONLY: Create a drop that includes new files added within the duration of the cool_off field.
  • TOP_LEVEL_FILES: Create a drop for each file placed in the top level of the source folder.
  • TOP_LEVEL_DIRS: Create a drop for each directory placed in the top level of the source folder. This drop also includes the sub-directories and files in the top level directory.
COOL_OFF_ONLY
cool_off

The duration allowed for new files to be included in a drop. This field is only relevant for the COOL_OFF_ONLY detection strategy. Aspera recommends choosing a multiple of the specified snapshot_creation_period for predictable results.

5s

Post Processing Fields

{
   ...
   "post_processing":{  
      "local":{  
         "type":"TRANSFER_NONE",
         "archive_dir":"C:\watchfolder_sessions\{$UUID}_{$DATETIME}"
      }
   },
   ...
}
Field Description Default
type
Determines whether files are archived, deleted, or kept after transfer of a drop.
  • TRANSFER_NONE: Files stay in the source directory.
  • TRANSFER_ARCHIVE: Files in the source directory are moved to a final archive after successful transfer.
  • TRANSFER_DELETE: Files in the source directory are deleted after successful transfer.
TRANSFER_NONE
archive_dir The destination of archived files, if the archive type is TRANSFER_ARCHIVE. The path can be determined using the following variables:
  • {$TIMESTAMP} (Drop creation time in seconds since epoch)
  • {$DAY_OF_MONTH} (Time format for drop's creation time)
  • {$MONTH}
  • {$YEAR}
  • {$HOUR}
  • {$MINUTE}
  • {$SECOND}
  • {$DATETIME} (alias for {$YEAR}{$MONTH}{$DAY_OF_MONTH}-{$HOUR}{$MINUTE}{$SECOND})
  • {$UUID}
  • {$NAME}
  • {$STATE}
  • {$FILE::STATE} (such as SUCCEEDED, FAILED)
N/A

Filter Fields

{
   ...
   "filters":[
      {
         "type":"GLOB",
         "pattern":"*.txt",
         "rule":"EXCLUDE"
      },
      {
         "type":"REGEX",
         "pattern":".*\\.txt",
         "rule":"INCLUDE"
      }
   ],
   ...
}
Field Description Default
type

The type of filter. Supported filters are GLOB and REGEX.

N/A
pattern The filter pattern. N/A
rule The rule for the filter. Supported rules are INCLUDE and EXCLUDE. N/A

Packages Fields

{
   ...
   "packages":{
      "timeout":"10s",
      "parsers":[
         {
            "final_transfer":"LIST",
            "filters":[
               {
                  "type":"REGEX",
                  "rule":"INCLUDE",
                  "pattern":".*\\.txt"
               }
            ],
            ...
         }
      ]
   },
   ...
}
Field Description Default
final_transfer Defines which file has to be transferred last.
  • LIST: The last file in the package list is transferred last.
  • LAST_FILE_IN_LIST: The files are transferred without any specific order.
LIST
filters A list of filters as described in the Filter Fields section above. N/A

The transport object

}
   ...
   "transport":{  
      "host":"198.51.100.22",
      "user":"aspx2",
      "pass":"XF324cd28",
      "token":"fiewle535etn23TEIW234n5sEWTnseonts",
      "proxy":"dnat://aspx1:XF324cd28@localhost:9001",
      "keypath":"~/.ssh/id_rsa",
      "fingerprint":"stringalsdjkfad",
   ...
   }
}
Option Description Default
host The hostname of the destination node. N/A
user The SSH account login name. N/A
pass The SSH account password. N/A
token The token key string. Not valid for use with growing files. N/A
proxy The address of an Aspera high-speed proxy server. The proxy syntax is: dnat(s)://user@server:port N/A
keypath The path to the private key file used for public key authentication.
Note: If a relative path is provided, the file at the relative path is checked for existence. If the relative path is not found, $HOME/.ssh/ is prepended to the relative path.
N/A
fingerprint Fingerprint to check against server SSH host key fingerprint N/A

Error Handling Fields

}
   ...
   "transport":{  
      ...
      "error_handling":{  
         "file":{  
            "max_retries":3,
            "retry_timeout":"3s"
         },
         "drop":{  
            "retry_period":"1m"
         }
      },
   ...
   }
}

Watch folder error handling distinguishes between two different error categories:

File-Specific Errors: These errors increase the file retry count every time a failure occurs. When the max_retries count is reached, the file is marked as failed and the session attempts to transfer the next file in the drop queue. All errors except the following:
  • License error
  • Authentication error
  • Any other error in establishing an ascp session

Other Errors: These errors do not increase the file retry count. If a given error re-occurs again and again, the same file is retried until the drop’s retry_period is exceeded. Then, the drop is marked as failed.

Option Description Default
max_retries When the maximum retry count is reached, the file is marked as failed. 3
retry_timeout Retry attempts are spaced by the timeout parameter. 3s
retry_period If no bytes are transferred during this period and no file is completed, the drop and all remaining incomplete files of the drop are marked as failed. 1m

The regular object (for ascp Sessions)

{
   ...
   "transport":{
      ...
      "regular":{  
         "max_parallel":10,
         "connect_timeout":"10s",
         "policy":"FAIR",
         "min_rate":"0B",
         "target_rate":"10M",
         "tcp_port":22,
         "udp_port":33001,
         "read_blk_size":"",
         "write_blk_size":"",
         "datagram_size":"",
         "rexmsg_size":"",
         "cipher":"AES128",
         "overwrite":"DIFF",
         "resume":"NONE",
         "preserve_uid":false,
         "preserve_gid":false,
         "preserve_time":false,
         "preserve_creation_time":false,
         "preserve_modification_time":false,
         "preserve_access_time":false,
         "queue_threshold":"5s",
         "sample_period":"2s",
         "raw_options":["-L","/tmp/log"],
         "content_protect_password":"ear_password"
      },
      ...  
   }
}
Field Description Default
max_parallel The maximum number of concurrent ascp sessions the watch folder can initiate. 10
connect_timeout Timeout duration in seconds before considering ascp has failed. 10s
policy Defines how ascp manages the bandwidth. The policy can be set to the following values:
  • FIXED
  • FAIR
  • HIGH
  • LOW
FAIR
min_rate Minimum rate in bytes that ascp can use. 0B
target_rate The target rate in megabytes that ascp should use. 10M
tcp_port The TCP port used for the FASP session. 22
udp_port The UPD port used by FASP for data transfer. 33001
read_blk_size The read block size. Default determined by settings in aspera.conf
write_blk_size The write block size. Default determined by settings in aspera.conf
datagram_size The datagram size (MTU) for FASP. Uses the detected path MTU.
rexmsg_size The maximum size of a retransmission request. Determined by ascp
cipher The encryption cipher for file data. You can chooose AES128 or NONE. AES128
overwrite Determines if a file is overwritten if it already already exists at the destination. You can choose:
  • NEVER
  • ALWAYS
  • DIFF
  • OLDER
  • DIFF+OLDER
DIFF
resume Determine if partial transfers are resumed.
  • NONE: Always transfer the entire file
  • FILE_ATTRIBUTES: Resume if file attributes match.
  • SPARSE_CHECKSUM: Resume if file attributes and sparse checksum match.
  • FULL_CHECKSUM: Resume if file attributes and full checksum match.
NONE
preserve_uid Preserve the file owner user ID. false
preserve_gid Preserve the file owner group ID. false
preserve_time This option is equivalent to configuring preserve_creation_time, preserve_modification_time, and preserve_access_time. false
preserve_creation_time Set creation time of the destination be set to that of the source. If the destination is a non-Windows host, this option is ignored. false
preserve_modification_time Set the modification time of the destination file be set to that of the source. false
preserve_access_time Set the access time of the destination to be set to that of the source. The destination file ends up with the access time of the source file prior to the transfer. false
queue_threshold Watch folder controls the amount of data pushed to ascp for transferring. When the capacity is reached, the watch folder waits before pushing new data. This capacity is based on the effective bandwidth reported by ascp. 5s
sample_period Period used to compute the current bandwidth. Used with queue_threshold to compute the amount of data pushed to ascp. 2s
raw_options Enable the use of new ascp options that are not yet available in the configuration. Optionally set logging of raw options.
Note: This option requires that raw options are also enabled in aspera.conf.
disabled
content_protect_password Provide the encryption-at-rest password. N/A

The growing Object (for faspstream Sessions)

{
   ...
   "transport":{
      ...
      "growing_file":{  
         "max_parallel":8,
         "policy":"FAIR",
         "min_rate":"",
         "target_rate":"10M",
         "tcp_port":22,
         "udp_port":33001,
         "datagram_size":"",
         "cipher":"AES128",
         "completion_timeout":"5s",
         "memory":"2M",
         "chunk_size":"128K",
         "force_send_after":"2s",
         "filters":[  
            {
              "type":"REGEX",
              "rule":"INCLUDE",
              "pattern":".*\\.growing"
            }
         ]
      }
   }
}
Option Description Default
max_parallel The maximum number of concurrent faspstream sessions the watch folder can initiate. 8
policy Defines how faspstream manages the bandwidth. The policy can be set to the following values:
  • FIXED
  • FAIR
  • HIGH
  • LOW
FAIR
min_rate Minimum rate in bytes that faspstream can use. 0B
target_rate The target rate in megabytes that faspstream should use. 10M
tcp_port The TCP port used for the session. 22
udp_port The UPD port used for data transfer. 33001
datagram_size The datagram size (MTU) for FASP. Uses the detected path MTU.
cipher The encryption cipher for file data. You can chooose AES128 or NONE. AES128
completion_timeout The duration before the session is considered complete. A growing file is considered completed when there is no new data for the duration of this timeout. 20s
force_send_after Force faspstream to send data after the given time, even if the chunk is not full. 2s
memory

The maximum amount of memory the faspstream binary is allowed to use.

2M
chunk_size Packet size for transfers over the network. 128K
filters A file is a growing file if the filename matches the given filter. N/A