Applying Filters to Include and Exclude Files

Filters allow you to refine the list of files (or directories) designated for transfer. With filters, you indicate which files in the transfer list to skip or include. At runtime, ascp looks for filters in two locations: on the ascp command line, and in aspera.conf. Filters can be set in the aspera.conf file either from the GUI, or by modifying it directly with an editor or asconfigurator. When filtering rules are found in aspera.conf, they are applied before rules on the command line. If no filtering rules are specified, ascp transfers all source files in the transfer list. This topic describes filtering using option flags on the ascp command line.

Note: Filter settings apply only when the server is acting as a client. Servers cannot exclude files or directories uploaded or downloaded by remote clients.

Specifying Rules on the Command Line

To specify filtering rules on the ascp command line, use the -E and -N options:

  • -E pattern      Exclude files or directories matching pattern.
  • -N pattern      Include files or directories matching pattern.

Each rule consists of a -E or -N option and its pattern. A pattern can be a file or directory name, or a set of names expressed with UNIX glob patterns.

To determine which files to transfer, each file in the set of source files to transfer (the transfer list) is evaluated by the filters as follows:
  1. ascp compares the next file (or directory) in the transfer list to the first rule.
  2. If the file matches the pattern, ascp includes it (-N) or excludes it (-E) and for this file, filtering stops.
  3. If the file does not match, ascp compares it with the next rule and repeats the process for each rule until a match is found or until all rules have been tried.
  4. If the file never matches any rules, it is included in the transfer.

Filtering operates only on the set of files and directories in the transfer list. That is, an include option (-N) cannot add files or directories that are not already part of the transfer list.

Filtering is a process of exclusion, and -N rules act as overrides to any -E rules that follow them. For example, consider the following example command:

$ ascp -N 'file2' -E 'file[0-9]' /tmp/L/file* user1@examplehost:/tmp
The transfer set is file* (all files that start with file). If file1, file2, and fileA are in /tmp/L, they are filtered as follows:
  1. When file1 is compared with the first rule (-N), no match is found, and filtering continues. When file1 is compared with the second rule (-E), there is a match; file1 is therefore excluded from transfer, and filtering stops for file1.
  2. When file2 is compared with the first rule, there is a match; file2 is therefore included in the transfer, and filtering stops for file2.
  3. When fileA is compared with the first rule, no match is found. When it is compared with the second rule, again no match is found. Because no further rules exclude it, fileA is therefore included in the transfer.

If directories or files reside in directories that have already been excluded, they will also be excluded and therefore not checked against any further rules. Thus, with the command-line options -E '/above/' -N '/above/below', the file /above/below is never considered because its parent directory /above/ has already been excluded.

Creating Rule Patterns

In order to filter directories and files to be transferred, their names are matched against patterns (globs) that include wildcards and special characters. The patterns use the standard globbing syntax found in UNIX systems as well as several Aspera extensions to the standard.

Character case: Case always matters, even if the scanned file system does not enforce such a distinction. For example, "debug" does not match "Debug". To match both, the pattern should be "[Dd]ebug".

Single quotes: Patterns must be interpreted only by ascp, not by the command shell. For this reason, patterns that contain wildcards should be surrounded by single quotes to protect them from expansion by the shell. (Even if patterns contain no wildcards, they can still be surrounded by single quotes.)

Partial matches: With globs, unlike standard regular expressions, the entire filename or directory name must match the pattern. That is, abcdef matches the pattern abc*f but abcdefg does not.

Pattern position: A pattern given with -N will match a path only if it falls directly under the transfer directory. However, a pattern given with -E will match a path regardless of where (which level) the path falls under the transfer directory. For example, given the pattern 'zzz*' and a transfer directory AAA:
  • The -N option matches only if the path to file (or directory) zzz falls directly under AAA. That is, AAA/zzz.
  • The -E option matches regardless of the where the path to file (or directory) zzz falls under AAA. For example, AAA/abc/def/zzz.

Standard Globbing: Wildcards and Special Characters

/ The only recognized path separator.
\ Quotes any character literally, including itself. The \ character is exclusively a quoting operator, not a path separator.
* Matches zero or more characters, except a / , or the . when preceded immediately by a / character.
? Matches any single character, except a / , or a . when preceded immediately by a / character.
[] Matches exactly one of a set of characters, except a / or a . preceded immediately by a / character.
[^] When ^ is the first character, matches exactly one character not in the set.
[!] When ! is the first character, matches exactly one character not in the set.
[x-x] Matches exactly one of a range of characters.
[:xxxxx:] For details about this type of wildcard, see any POSIX-standard guide to globbing.

Globbing Extensions: Wildcards and Special Characters

/** Like * but also matches the / character, or a . preceded immediately by a / (that is, the . in /. ).
* or /** at end of pattern Matches both directories and files.
/ at end of pattern Matches directories only. With -N, no files under matched directories or their subdirectories are included in the transfer. All subdirectories are still included, although their files will not be included. However, with -E, excluding a directory also excludes all files and subdirectories under it.
no / or * at end of pattern Matches files only.
/ at start of pattern Must match the entire string from the root of the transfer set. (Note: The leading / does not refer to the system root or the docroot.)

Standard Globbing Examples

Wildcard Example Matches Does Not Match
/ abc/def/xyz abc/def/xyz abc/def
\ abc\? abc? abc\? abc/D abcD
* abc*f abcdef abc.f abc/f abcefg
? abc?? abcde abc.z abcdef abc/d abc/.
[] [abc]def adef cdef abcdef ade
[^] [^abc]def zdef .def 2def bdef /def /.def
[!] [!abc]def zdef .def 2def cdef /def /.def
[:xxxxx:] [[:lower:]]def cdef ydef Adef 2def .def

Globbing Extension Examples

Wildcard Example Matches Does Not Match
/** a/**/f a/f a/.z/f a/d/e/f a/d/f/ za/d/f
* at end of rule abc* abc/ abcfile  
/** at end of rule abc/** abc/.file abc/d/e/ abc/
/ at end of rule abc/*/ abc/dir abc/file
no / at end of rule abc abc (file) abc/
/ at start of rule /abc/def /abc/def xyz/abc/def

Rule Composition

Example Transfer Result
-N rule Includes all files and directories whose names match rule. Because there is no -E, all the originally specified files and directories are included anyway; in other words, by itself, a -N rule does nothing.
-N rule1 -E rule2 Includes all files and directories whose names match rule1. Excludes all that match rule2, except those that also matched rule1.
-E rule Excludes all files and directories whose names match rule.
-E rule1 -N rule2 Excludes all files and directories whose names match rule1. Because there is no -E following the -N, all files and directories not already excluded by the preceding -E are included anyway; in other words, a trailing -N rule does nothing to change the result.

Testing Your Filter Rules

If you plan to use filtering rules, it's best to test them first. An easy way to test filtering rules, or to learn how they work, is to set up source and destination directories and use demo.asperasoft.com as the Aspera server:

  1. On your computer, create a small set of directories and files that generally matches a file set you typically transfer. Since filenames are all that matter, the files can be small.
  2. Place the file set in an accessible location, for example /tmp/src.
  3. Upload the file set to the Aspera demo server as user "aspera". Specify the demo-server target directory Upload. You will be prompted for the password, which is "demoaspera":
    $ ascp /tmp/src aspera@demo.asperasoft.com:Upload/
  4. Create a destination directory on your computer, for example /tmp/dest.
  5. You can now download your files from the demo server to /tmp/dest, running the ascp commands with -N and -E to test your filtering rules. For example:
    $ ascp -N 'wxy/**' -E 'def' aspera@demo.asperasoft.com:Upload/src/abc/ /tmp/dest
  6. Compare the destination directory with the source to determine whether files were filtered as expected.
    $ diff -r dest/ src/

    The diff output will show the missing (untransferred) files and directories.

Example Filter Rules

The example rules below are based on running a command such as the following to download a directory AAA from demo.asperasoft.com to /tmp/dest:

$ ascp rules aspera@demo.asperasoft.com:Upload/AAA /tmp/dest

The examples below use the following file set:

AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile

Key for interpreting example results below:

< xxx/yyy = Excluded
xxx/yyy = Included
zzz/ = directory name
zzz = filename

(1) Transfer everything except files and directories starting with ".":

-N '*' -E 'AAA/**'

Results:

AAA/abc/def
AAA/abc/wxy/def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/.def

(2) Exclude directories and files whose names start with wxy:

-E 'wxy*' 

Results:

AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/xyz/def/
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/abc/xyz/def/wxy
< AAA/wxyfile
< AAA/wxy/xyx/
< AAA/wxy/xyxfile

(3) Include directories and files that start with "wxy" if they fall directly under AAA:

-N 'wxy*' -E 'AAA/**'

Results:

AAA/wxy/
AAA/wxyfile
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/abc/xyz/def/wxy
< AAA/wxy/xyx/
< AAA/wxy/xyxfile

(4) Include directories and files at any level that start with wxy, but do not include dot-files, dot-directories, or any files under the wxy directories (unless they start with wxy). However, subdirectories under wxy will be included:

-N '*/wxy*' -E 'AAA/**' 

Results:

AAA/abc/wxy/tuv/
AAA/abc/xyz/def/wxy
AAA/wxyfile 
AAA/wxy/xyx/
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/def     * 
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/wxy/xyxfile

* Even though wxy is included, def is excluded because it's a file.

(5) Include wxy directories and files at any level, even those starting with ".":

-N '*/wxy*' -N '*/wxy/**' -E 'AAA/**'

Results:

AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def

(6) Exclude directories and files starting with wxy, but only those found at a specific location in the tree:

-E '/AAA/abc/wxy*' 

Results:

AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def

(7) Include the wxy directory at a specific location, and include all its subdirectories and files, including those starting with ".":

-N 'AAA/abc/wxy/**' -E 'AAA/**' 

Results:

AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/xyz/def/wxy
< AAA/wxyfile
< AAA/wxy/xyx/
< AAA/wxy/xyxfile