ascp: Transferring from the Command Line |
Filters allow you to refine the list of files (or directories) designated for transfer. With filters, you indicate which files in the transfer list to skip or include. At runtime, ascp looks for filters in two locations: on the ascp command line, and in aspera.conf. Filters can be set in the aspera.conf file either from the GUI, or by modifying it directly with an editor or asconfigurator. When filtering rules are found in aspera.conf, they are applied before rules on the command line. If no filtering rules are specified, ascp transfers all source files in the transfer list. This topic describes filtering using option flags on the ascp command line.
To specify filtering rules on the ascp command line, use the -E and -N options:
Each rule consists of a -E or -N option and its pattern. A pattern can be a file or directory name, or a set of names expressed with UNIX glob patterns.
Filtering operates only on the set of files and directories in the transfer list. That is, an include option (-N) cannot add files or directories that are not already part of the transfer list.
Filtering is a process of exclusion, and -N rules act as overrides to any -E rules that follow them. For example, consider the following example command:
$ ascp -N 'file2' -E 'file[0-9]' /tmp/L/file* user1@examplehost:/tmp
If directories or files reside in directories that have already been excluded, they will also be excluded and therefore not checked against any further rules. Thus, with the command-line options -E '/above/' -N '/above/below', the file /above/below is never considered because its parent directory /above/ has already been excluded.
In order to filter directories and files to be transferred, their names are matched against patterns (globs) that include wildcards and special characters. The patterns use the standard globbing syntax found in UNIX systems as well as several Aspera extensions to the standard.
Character case: Case always matters, even if the scanned file system does not enforce such a distinction. For example, "debug" does not match "Debug". To match both, the pattern should be "[Dd]ebug".
Single quotes: Patterns must be interpreted only by ascp, not by the command shell. For this reason, patterns that contain wildcards should be surrounded by single quotes to protect them from expansion by the shell. (Even if patterns contain no wildcards, they can still be surrounded by single quotes.)
Partial matches: With globs, unlike standard regular expressions, the entire filename or directory name must match the pattern. That is, abcdef matches the pattern abc*f but abcdefg does not.
Standard Globbing: Wildcards and Special Characters
/ | The only recognized path separator. |
\ | Quotes any character literally, including itself. The \ character is exclusively a quoting operator, not a path separator. |
* | Matches zero or more characters, except a / , or the . when preceded immediately by a / character. |
? | Matches any single character, except a / , or a . when preceded immediately by a / character. |
[ … ] | Matches exactly one of a set of characters, except a / or a . preceded immediately by a / character. |
[^… ] | When ^ is the first character, matches exactly one character not in the set. |
[!… ] | When ! is the first character, matches exactly one character not in the set. |
[x-x] | Matches exactly one of a range of characters. |
[:xxxxx:] | For details about this type of wildcard, see any POSIX-standard guide to globbing. |
Globbing Extensions: Wildcards and Special Characters
/** | Like * but also matches the / character, or a . preceded immediately by a / (that is, the . in /. ). |
* or /** at end of pattern | Matches both directories and files. |
/ at end of pattern | Matches directories only. With -N, no files under matched directories or their subdirectories are included in the transfer. All subdirectories are still included, although their files will not be included. However, with -E, excluding a directory also excludes all files and subdirectories under it. |
no / or * at end of pattern | Matches files only. |
/ at start of pattern | Must match the entire string from the root of the transfer set. (Note: The leading / does not refer to the system root or the docroot.) |
Standard Globbing Examples
Wildcard | Example | Matches | Does Not Match |
---|---|---|---|
/ | abc/def/xyz | abc/def/xyz | abc/def |
\ | abc\? | abc? | abc\? abc/D abcD |
* | abc*f | abcdef abc.f | abc/f abcefg |
? | abc?? | abcde abc.z | abcdef abc/d abc/. |
[ … ] | [abc]def | adef cdef | abcdef ade |
[^… ] | [^abc]def | zdef .def 2def | bdef /def /.def |
[!… ] | [!abc]def | zdef .def 2def | cdef /def /.def |
[:xxxxx:] | [[:lower:]]def | cdef ydef | Adef 2def .def |
Globbing Extension Examples
Wildcard | Example | Matches | Does Not Match |
---|---|---|---|
/** | a/**/f | a/f a/.z/f a/d/e/f | a/d/f/ za/d/f |
* at end of rule | abc* | abc/ abcfile | |
/** at end of rule | abc/** | abc/.file abc/d/e/ | abc/ |
/ at end of rule | abc/*/ | abc/dir | abc/file |
no / at end of rule | abc | abc (file) | abc/ |
/ at start of rule | /abc/def | /abc/def | xyz/abc/def |
Example | Transfer Result |
---|---|
-N rule | Includes all files and directories whose names match rule. Because there is no -E, all the originally specified files and directories are included anyway; in other words, by itself, a -N rule does nothing. |
-N rule1 -E rule2 | Includes all files and directories whose names match rule1. Excludes all that match rule2, except those that also matched rule1. |
-E rule | Excludes all files and directories whose names match rule. |
-E rule1 -N rule2 | Excludes all files and directories whose names match rule1. Because there is no -E following the -N, all files and directories not already excluded by the preceding -E are included anyway; in other words, a trailing -N rule does nothing to change the result. |
If you plan to use filtering rules, it's best to test them first. An easy way to test filtering rules, or to learn how they work, is to set up source and destination directories and use demo.asperasoft.com as the Aspera server:
The example rules below are based on running a command such as the following to download a directory AAA from demo.asperasoft.com to /tmp/dest:
$ ascp rules aspera@demo.asperasoft.com:Upload/AAA /tmp/dest
The examples below use the following file set:
AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< xxx/yyy = Excluded
xxx/yyy = Included
zzz/ = directory name
zzz = filename
(1) Transfer everything except files and directories starting with ".":
-N '*' -E 'AAA/**'
Results:
AAA/abc/def AAA/abc/wxy/def AAA/abc/wxy/tuv/def AAA/abc/xyz/def/wxy AAA/wxyfile AAA/wxy/xyx/ AAA/wxy/xyxfile < AAA/abc/.def < AAA/abc/.wxy/def < AAA/abc/wxy/.def
(2) Exclude directories and files whose names start with wxy:
-E 'wxy*'
Results:
AAA/abc/def AAA/abc/.def AAA/abc/.wxy/def AAA/abc/xyz/def/ < AAA/abc/wxy/def < AAA/abc/wxy/.def < AAA/abc/wxy/tuv/def < AAA/abc/xyz/def/wxy < AAA/wxyfile < AAA/wxy/xyx/ < AAA/wxy/xyxfile
(3) Include directories and files that start with "wxy" if they fall directly under AAA:
-N 'wxy*' -E 'AAA/**'
Results:
AAA/wxy/ AAA/wxyfile < AAA/abc/def < AAA/abc/.def < AAA/abc/.wxy/def < AAA/abc/wxy/def < AAA/abc/wxy/.def < AAA/abc/wxy/tuv/def < AAA/abc/xyz/def/wxy < AAA/wxy/xyx/ < AAA/wxy/xyxfile
(4) Include directories and files at any level that start with wxy, but do not include dot-files, dot-directories, or any files under the wxy directories (unless they start with wxy). However, subdirectories under wxy will be included:
-N '*/wxy*' -E 'AAA/**'
Results:
AAA/abc/wxy/tuv/ AAA/abc/xyz/def/wxy AAA/wxyfile AAA/wxy/xyx/ < AAA/abc/def < AAA/abc/.def < AAA/abc/.wxy/def < AAA/abc/wxy/def * < AAA/abc/wxy/.def < AAA/abc/wxy/tuv/def < AAA/wxy/xyxfile
* Even though wxy is included, def is excluded because it's a file.
(5) Include wxy directories and files at any level, even those starting with ".":
-N '*/wxy*' -N '*/wxy/**' -E 'AAA/**'
Results:
AAA/abc/wxy/def AAA/abc/wxy/.def AAA/abc/wxy/tuv/def AAA/abc/xyz/def/wxy AAA/wxyfile AAA/wxy/xyx/ AAA/wxy/xyxfile < AAA/abc/def < AAA/abc/.def < AAA/abc/.wxy/def
(6) Exclude directories and files starting with wxy, but only those found at a specific location in the tree:
-E '/AAA/abc/wxy*'
Results:
AAA/abc/def AAA/abc/.def AAA/abc/.wxy/def AAA/abc/xyz/def/wxy AAA/wxyfile AAA/wxy/xyx/ AAA/wxy/xyxfile < AAA/abc/wxy/def < AAA/abc/wxy/.def < AAA/abc/wxy/tuv/def
(7) Include the wxy directory at a specific location, and include all its subdirectories and files, including those starting with ".":
-N 'AAA/abc/wxy/**' -E 'AAA/**'
Results:
AAA/abc/wxy/def AAA/abc/wxy/.def AAA/abc/wxy/tuv/def < AAA/abc/def < AAA/abc/.def < AAA/abc/.wxy/def < AAA/abc/xyz/def/wxy < AAA/wxyfile < AAA/wxy/xyx/ < AAA/wxy/xyxfile