Configuring Filters to Include and Exclude Files

Filters allow you to refine the list of files (or directories) designated for transfer. With filters, you indicate which files in the transfer list to skip or include. At runtime, ascp looks for filters in two locations: on the ascp command line, and in aspera.conf. When filtering rules are found in aspera.conf, they are applied before rules on the command line. Filters can be set in the aspera.conf file either from the GUI, or by modifying it directly with an editor or asconfigurator. If no filtering rules are specified, ascp transfers all source files in the transfer list. This topic describes how to configure filters from the GUI.

Note: Filter settings apply only when the server is acting as a client. Servers cannot exclude files or directories uploaded or downloaded by remote clients.

Specifying Filtering Rules from the File Handling Configuration Tab

To specify filtering rules from the GUI, click the Configuration button to open the Server Configuration dialog. Select the File Handling tab, and scroll down to File Filter Pattern List. Click the override checkbox to enable you to enter values to override any inherited settings for this option.



To specify filtering rules, click the button. This opens a Define dialog where you can type in filtering rules. If rules were added earlier, either through the GUI or through aspera.conf, they will appear in the window.



Click the plus sign in the top left corner button to add a new filtering rule, or click the minus sign in top left corner button to delete a rule that you've selected.

A rule consists of a "+" or "-" sign (indicating whether to include or exclude), followed by a space character, followed by a pattern. A pattern can be a file or directory name, or a set of names expressed with UNIX glob patterns (described below).

Note: Do not confuse the GUI's line-add and line-delete buttons plus sign in the top left corner and minus sign in top left corner with the include/exclude characters "+" or "-" that are part of rule syntax. The purpose of each is different and they are unrelated.
To determine which files to transfer, each file in the set of source files to transfer (the transfer list) is evaluated by the filters as follows:
  1. The first (or next) file or directory in the transfer list is compared to the first rule.
  2. If the file matches the pattern, it's included or excluded, and for this file, filtering stops.
  3. If the file does not match, it's compared with the next rule and the process is repeated for each rule until a match is found or until all rules have been tried.
  4. If the file did not match any rules, it is included in the transfer.

Filtering operates only on the set of files and directories in the transfer list. That is, an include option cannot add files or directories that are not already part of the transfer list.

Filtering is a process of exclusion, and "+" rules act as overrides to any "-" rules that follow them. For example, consider the following example rules when the transfer set is file*:

+ file2
- file[0-9]
The transfer set is all files that start with file). If file1, file2, and fileA are in /tmp/L, they are filtered as follows:
  1. When file1 is compared with the first rule (+), no match is found, and filtering continues. When file1 is compared with the second rule (-), there is a match; file1 is therefore excluded from transfer, and filtering stops for file1.
  2. When file2 is compared with the first rule, there is a match; file2 is therefore included in the transfer, and filtering stops for file2.
  3. When fileA is compared with the first rule, no match is found. When it is compared with the second rule, again no match is found. Because no further rules exclude it, fileA is therefore included in the transfer.
If directories or files reside in directories that have already been excluded, they will also be excluded and therefore not checked against any further rules. For example, consider the following rules:
- /above/ 
+ /above/below
The file /above/below is never considered because its parent directory /above/ has already been excluded.

Creating Rule Patterns

In order to filter directories and files to be transferred, their names are matched against patterns (globs) that include wildcards and special characters. The patterns use the standard globbing syntax found in UNIX systems along with several extensions.

Character case: Case always matters, even if the scanned file system does not enforce such a distinction. For example, "debug" does not match "Debug". To match both, the pattern should be "[Dd]ebug".

Partial matches: With globs, unlike standard regular expressions, the entire filename or directory name must match the pattern. That is, abcdef matches the pattern abc*f but abcdefg does not.

Pattern position: A pattern given with "+" will match a path only if it falls directly under the transfer directory. However, a pattern given with "-" will match a path regardless of where (at which level) the path falls under the transfer directory. For example, given the pattern 'zzz*' and a transfer directory AAA:
  • The "+" option matches only if the path to the zzz file (or directory) falls directly under AAA. That is, AAA/zzz.
  • The "-" option matches regardless of the where the path to the zzz file (or directory) falls under AAA. For example, AAA/abc/def/zzz.

Standard Globbing: Wildcards and Special Characters

/ The only recognized path separator.
\ Quotes any character literally, including itself. The \ character is exclusively a quoting operator, not a path separator.
* Matches zero or more characters, except a / , or the . when preceded immediately by a / character.
? Matches any single character, except a / , or a . when preceded immediately by a / character.
[] Matches exactly one of a set of characters, except a / or a . preceded immediately by a / character.
[^] When ^ is the first character, matches exactly one character not in the set.
[!] When ! is the first character, matches exactly one character not in the set.
[x-x] Matches exactly one of a range of characters.
[:xxxxx:] For details about this type of wildcard, see any POSIX-standard guide to globbing.

Globbing Extensions: Wildcards and Special Characters

/** Like * but also matches the / character, or a . preceded immediately by a / (that is, the . in /. ).
* or /** at end of pattern Matches both directories and files.
/ at end of pattern Matches directories only. With -N, no files under matched directories or their subdirectories are included in the transfer. All subdirectories are still included, although their files will not be included. However, with -E, excluding a directory also excludes all files and subdirectories under it.
no / or * at end of pattern Matches files only.
/ at start of pattern Must match the entire string from the root of the transfer set. (Note: The leading / does not refer to the system root or the docroot.)

Standard Globbing Examples

Wildcard Example Matches Does Not Match
/ abc/def/xyz abc/def/xyz abc/def
\ abc\? abc? abc\? abc/D abcD
* abc*f abcdef abc.f abc/f abcefg
? abc?? abcde abc.z abcdef abc/d abc/.
[] [abc]def adef cdef abcdef ade
[^] [^abc]def zdef .def 2def bdef /def /.def
[!] [!abc]def zdef .def 2def cdef /def /.def
[:xxxxx:] [[:lower:]]def cdef ydef Adef 2def .def

Globbing Extension Examples

Wildcard Example Matches Does Not Match
/** a/**/f a/f a/.z/f a/d/e/f a/d/f/ za/d/f
* at end of rule abc* abc/ abcfile  
/** at end of rule abc/** abc/.file abc/d/e/ abc/
/ at end of rule abc/*/ abc/dir abc/file
no / at end of rule abc abc (file) abc/
/ at start of rule /abc/def /abc/def xyz/abc/def

Rule Composition

Example Transfer Result
+ rule Includes all files and directories whose names match rule. Because there is no "-", all the originally specified files and directories are included anyway; in other words, by itself, a "+" rule does nothing.
- rule Excludes all files and directories whose names match rule.

+ rule1
- rule2

Includes all files and directories whose names match rule1. Excludes all that match rule2, except those that also matched rule1.

- rule1
+ rule2

Excludes all files and directories whose names match rule1. Because there is no "-" following the "+", all files and directories not already excluded by the preceding "-" are included anyway; in other words, a trailing "+" rule does nothing to change the result.

Testing Your Filtering Rules

If you plan to use filtering rules, it's best to test them first. An easy way to try filtering rules, or to learn how they work, is to set up source and destination directories and use demo.asperasoft.com as the Aspera server:

  1. On your computer, create a small set of directories and files that generally matches a file set you typically transfer. Since filenames are all that matter, the size of the files can be small.
  2. Place the file set in an accessible location, for example C:\tmp\src.
  3. Upload the file set to the Aspera demo server as user "aspera". Specify the demo-server target directory Upload. When you are prompted for the password, enter "demoaspera". For more information about setting up a connection to demo.asperasoft.com, see Testing a Locally Initiated Transfer.
  4. Create a destination directory on your computer, for example C:\tmp\dest.
  5. You can now download your files from the demo server to C:\tmp\dest and test your filtering rules. For example:

  6. Compare the list of files transferred to the list of your original files.

Example Filter Rules

The example rules below are based on downloading a directory AAA from demo.asperasoft.com to C:\tmp\dest on your computer:



The examples below use the following file set:

AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
Key for interpreting results:

< xxx/yyy = Excluded
xxx/yyy = Included
zzz/ = directory name
zzz = filename

(1) Transfer everything except files and directories starting with ".":

+ * 
- AAA/**

Results:

AAA/abc/def
AAA/abc/wxy/def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/.def

(2) Exclude directories and files whose names start with wxy

- wxy* 

Results:

AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/xyz/def/
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/abc/xyz/def/wxy
< AAA/wxyfile
< AAA/wxy/xyx/
< AAA/wxy/xyxfile

(3) Include directories and files that start with "wxy" if they fall directly under AAA:

+ wxy* 
- AAA/**

Results:

AAA/wxy/
AAA/wxyfile
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/abc/xyz/def/wxy
< AAA/wxy/xyx/
< AAA/wxy/xyxfile

(4) Include directories and files at any level that start with wxy, but do not include dot-files, dot-directories, or any files under the wxy directories (unless they start with wxy). However, subdirectories under wxy will be included:

+ */wxy*
- AAA/** 

Results:

AAA/abc/wxy/tuv/
AAA/abc/xyz/def/wxy
AAA/wxyfile 
AAA/wxy/xyx/
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/def     * 
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/wxy/xyxfile

* Even though wxy is included, def is excluded because it's a file.

(5) Include wxy directories and files at any level, even those starting with ".":

+ */wxy*
- */wxy/**
- AAA/**

Results:

AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def

(6) Exclude directories and files starting with wxy, but only those found at a specific location in the tree:

+ /AAA/abc/wxy* 

Results:

AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def

(7) Include the wxy directory at a specific location, and include all its subdirectories and files, including those starting with ".":

+ AAA/abc/wxy/**
- AAA/** 

Results:

AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/xyz/def/wxy
< AAA/wxyfile
< AAA/wxy/xyx/
< AAA/wxy/xyxfile