Remote File Watcher Action Plugin

Overview

This action plugin waits for the arrival of files matching a certain pattern on a remote Aspera server.

Saved Parameters Description

Name: The name used to identify a saved RemoteFileWatcher configured instance.
Comments: Some comments about this saved RemoteFileWatcher configured instance.
Remote Node: The remote node name from defined Remote Nodes.
Watch directory: The directory search path for files or folders.
Multi-folder search?: If selected, indicates that a list of folder will be provided. If the folders are set at design time, they can be entered either using a semi-colon for separator, or alternatively the ruby array notation can be used (ie.: ['folder1','folder2'])
Descend in sub directories?: If selected, indicates that the watchfolder will look for file on the base path and in all subfolders at all levels.
File pattern: The detection file name or pattern expressed as a regular expression.
Ignore folders?: Folders will be ignored. Only matching files are returned.
Ignore instant files?: Ignore files for which the creation date equals the last modification date (condition true for some Windows files that are still growing).
Ignore zero-byte files?: Ignore empty files (files whose size is 0 bytes)
Polling frequency: Interval in seconds to search for matching files in the watch directory.
Check only once: Check for matching file presence (do not wait) and stop processing.
Keep on-going: Wait for matching file(s), spawn a new workflow instance each time matching file(s) are detected and proceed with the next steps. The new instance will wait for new matching files. If this option is not set, wait for matching files once and proceed with the next steps.
Return single match?: Return only one matching file at a time. Otherwise return matching files as an array.
Maximum number of files detected: Limits the maximum number of files detected in a single round. Use 0 to specify no limit.
Trigger on partial: A matching file will be detected even if not stable.
Stability cool-off time: Optional - The minimum time it takes for a file to be seen unchanged to be considered stable. If not provided, the polling frequency is used.
Scan time out: Increase the scan time out value for folders with large number of files
Enable extensive debug: If set it provides extensive debug information when listing files on remote node. Must be used only in debugging situations.
Force exact matching: This option disable regex matching for the file name. The watcher will only detect files that match exactly the File pattern.
Trigger persistence scope: Sets the scope/availability of the trigger output. The default value is 'by workorder group'. The definitions are: 'by workorder group', 'by action template', 'absolute', 'by step', 'by workflow', 'none'

Details:

'by workorder group' - Trigger files are tracked while the workorder is executing and will only be trapped once for a workorder group. If the workorder is restarted (new workorder group), all past triggers will be detected again if still matching the search criteria.
'by workflow' - Same as 'by workorder group' except no triggering will occur upon a 'restart' action for the same workflow.
'by action template' - Allows the tracking and thus non-triggering in another workflow for previously processed trigger files shared across workflows via the same template.
'absolute' - The trigger list is tracked regardless of where it occurs.
'by step' - Similar to 'by workflow', but allows for multiple independent trigger steps within the workflow.
'none' - No persistence at all. If a past trigger file is still present, it will re-trigger if 'keep on-going' is selected. This option is recommended when 'keep on-going' is not selected.

Inputs Description

The list of inputs depends on the configuration of the RemoteFileWatcher action template.

Changes in this list of inputs will override any values set in the action template.

Watch_directory: The directory search path for files or folders.
File_pattern: The detection file name or pattern
remote_node_user: Remote node user name
remote_node_password: Remote node password
remote_node: Remote node IP address or domain name

Outputs Description

File_list: The matching files as an array
File_path: A single matching file as an string

Supported Actions

None

Dependencies

None

Operating Instructions

Overview:

This action detects new files or folders matching a certain pattern, located below a watch directory in a remote server. Both the pattern matching and the directory path must be provided by the user. The directory may be searched a single time for a matching file (do not select keep on-going) or can be treated similarly to a hot folder where it is continually scanned for matching files (select keep on-going). Matching file presence can also be checked (select check only once), going on the failure path if no matching file is found.

Instructions:

Populate the watch directory field with the directory to be scanned for trigger files. Globbing pattern matching is used in watch directory.

If the trigger file can appear in a sub directory directly below this "root" directory, then it is important to place a wild card in the Watch directory name to tell the system that it is necessary to traverse through the sub folder to search for the file. For example, to traverse the first directory the path could be something like "/foo/bar/*". This will instruct the system to search the first directory below "bar" for the trigger file (e.g. detecting /foo/bar/toto.txt).

To detect files one more level down the path, you can use the Descend in sub directories? flag. If the trigger file can only appear in the root directory (or in a given directory), it is not advantageous to use the Descend in sub directories? flag for performance reasons.

Other example: /fo? will match /foo or /fo1 but not /foo1

The search is case sensitive.

Populate the file pattern field with the file name or file pattern used for the trigger.

Regular pattern matching is used for file pattern. Entering .*\.txt will result in all files with the .txt extension being identified as a trigger. Multiple patterns can be entered with (.*\.mxf|.*\.mpg)

It works for remote nodes installed with an Aspera transfer server (for example, Point to Point) as it uses the Aspera ascmd executable to browse files. The remote node must run an SSH server because SSH is used for authentication purposes. The remote user name must be an operating system account that must be defined in /opt/aspera/etc/aspera.conf on the remote server. The path to configure in the watch directory is relative to the remote user docroot as declared in /opt/aspera/etc/aspera.conf such as:

                       <user> 
                        <name>xfer</name> 
                        <file_system> 
                            <access> 
                                <paths> 
                                    <path> 
                                        <absolute>/home/xfer/data</absolute> 
                                    </path> 
                                </paths> 
                            </access> 
                        </file_system> 
                        </user>

For example, a watch directory like "/foo" refers to "/home/xfer/data/foo" on the remote server. Also the files/folders need to have the right permissions for the given remote user in order to be detected.

Select the checkbox "Keep trigger on-going" if it is necessary to periodically scan for files and folders arriving into the Watch directory and act upon their arrival. "Check only once" and "Keep trigger on-going" are incompatible options (do not select both of them).

If it is OK to trigger on a partial file, select "Triggers on partial file". This will allow processing to begin before the trigger file has been completely updated.