aspera.conf - Filters to Include and Exclude Files

Filters allow you to refine the list of files (or directories) designated for transfer. With filters, you indicate which files in the transfer list to skip or include. At runtime, ascp looks for filters in two locations: on the ascp command line, and in aspera.conf. Filters can be set in the aspera.conf file by modifying it directly with an editor or asconfigurator. When filtering rules are found in aspera.conf, they are applied before rules on the command line. If no filtering rules are specified, ascp transfers all source files in the transfer list. This topic describes how to specify rules by editing aspera.conf or using asconfigurator.

Note: Filter settings apply only when the server is acting as a client. Servers cannot exclude files or directories uploaded or downloaded by remote clients.

Setting Filter Rules in aspera.conf

The ascp -N and -E options let you specify filter rules individually for each transfer, while filter options configured in aspera.conf allow you to have the same rules applied to all transfers.

Filter rules that ascp finds in aspera.conf are always applied before any command-line rules. This allows you to specify individual command-line rules to augment a core set specified in aspera.conf.

Filter rules can be set in aspera.conf in the following ways:

  • by modifying aspera.conf directly with a text editor
  • by modifying aspera.conf with the asconfigurator tool.

This section describes how to use the asconfigurator method. However, any of the examples can also be created by editing aspera.conf directly.

Below is a section of an aspera.conf file showing settings for filter rules. In this example, the filter rules are in the <default> section, which is where global options are set:

    <default>
        <file_system>
            <filters>
                <filter>+ file.txt</filter>
		  <filter>- *.txt</filter>
            </filters>
        </file_system>
    </default>

Each rule is specified with a separate <filter> option. A rule consists of a "+" or "-" sign (indicating whether to include or exclude), followed by a space character, followed by a pattern. A pattern can be a file or directory name, or a set of names expressed with UNIX glob patterns (described below). (The equivalent ascp command options for the above example are: -N 'file.txt' -E '*.txt' )

To determine which files to transfer, each file in the set of transfer files (the transfer list) is evaluated by the filter rules in top-down order, as follows:
  1. The first (or next) file or directory in the transfer list is compared to the first rule.
  2. If the file matches the pattern, it's included (if +) or excluded (if -), and for this file, filtering stops.
  3. If the file does not match, it's compared with the next rule and the process is repeated for each rule until a match is found or until all rules have been tried.
  4. If the file did not match any rules, it is included in the transfer.

Filtering is a process of exclusion, and "+" rules act as overrides to any "-" rules that follow them.

Filtering operates only on the set of files and directories in the transfer list. That is, an include option cannot add files or directories that are not already part of the transfer list.

If directories or files reside in directories that have already been excluded, they will also be excluded and therefore not checked against any further rules. For example, consider the following rules:
- /above/ 
+ /above/below
The file /above/below is never considered because its parent directory /above/ has already been excluded.

Creating Rule Patterns

In order to filter directories and files to be transferred, their names are matched against patterns (globs) that include wildcards and special characters. The patterns use the standard globbing syntax found in UNIX systems along with several extensions.

Character case: Case always matters, even if the scanned file system does not enforce such a distinction. For example, "debug" does not match "Debug". To match both, the pattern should be "[Dd]ebug".

Partial matches: With globs, unlike standard regular expressions, the entire filename or directory name must match the pattern. That is, abcdef matches the pattern abc*f but abcdefg does not.

Pattern position: A pattern given with "+" will match a path only if it falls directly under the transfer directory. However, a pattern given with "-" will match a path regardless of where (at which level) the path falls under the transfer directory. For example, given the pattern 'zzz*' and a transfer directory AAA:
  • The "+" option matches only if the path to the zzz file (or directory) falls directly under AAA. That is, AAA/zzz.
  • The "-" option matches regardless of the where the path to the zzz file (or directory) falls under AAA. For example, AAA/abc/def/zzz.

For details on using wildcards and special characters to build rule patterns, see Applying Filters to Include and Exclude Files.

Running Asconfigurator

In order to run asconfigurator successfully, you must (1) have write access to aspera.conf, and (2) not be executing it from aspshell, which does not allow running asconfigurator.

The set commands for user, group, and global filter settings use the following syntax:

asconfigurator -x "set_user_data;user_name,username;file_filters,|rule1|rule2...|ruleN"
asconfigurator -x "set_group_data;group_name,groupname;file_filters,|rule1|rule2...|ruleN"
asconfigurator -x "set_node_data;file_filters,|rule1|rule2...|ruleN"

Each rule argument, including the first, must begin with a "|" character, which serves as the separator between multiple rules.

To clear rules, run asconfigurator by specifying "file_filters," without rule arguments. Note that the comma in "file_filters," is still required. See the example below.

Note: Running asconfigurator replaces the specified settings; it does not add to them.

Examples

The following example creates the aspera.conf <default> code shown above:

$ asconfigurator -x "set_node_data;file_filters,|+ file.txt|- *.txt"

The following example sets filters for user asp1, producing the result below:

# asconfigurator -x "set_user_data;user_name,asp1;file_filters,|+ abc/wxy/tuv/**|- abc/**/def"
    <aaa>
        <realms>
            <realm>
                <users>
                    <user>
                        <name>asp1</name>
                        <file_system>
                            <filters>
                                <filter>+ abc/wxy/tuv/**</filter>
                                <filter>- abc/**/def</filter>
                            </filters>
                        </file_system>
                    </user>
                </users>
            </realm>
        </realms>
    </aaa>

The following example clears all filters for the group asgroup, producing the result below:

# asconfigurator -x "set_group_data;group_name,asgroup;file_filters,"

    <groups>
        <group>
            <name>asgroup</name>
            <file_system>
                <filters />
            </file_system>
        </group>
    </groups>

For further information on the use of asconfigurator, see Asconfigurator Reference.