Include/Exclude Filtering Rules

Notes on Separators and Case

You can synchronize Windows, Linux, Mac OS X, and other Unix-based endpoints and servers, but must take care with path separators. The path separator "/" is supported on Windows and other platforms. The path separator "\" is platform-agnostic only for the options -d/r/L/R/B/b and --keep-dir-local/remote. In Include/Exclude Filtering Rules, however, "\" is exclusively a quoting operator and "/" is the only path separator recognized.
Case always matters, even if the scanned file system does not enforce such a distinction. For example, on Windows FAT / NTFS file systems and Mac OS X HPFS+, if the user writes "DEBUG", it matches files "Debug" and "debug". In filter rules, however, the comparison is always exact. To pick up both "Debug" and "debug", the pattern must be "[Dd]ebug".

Specifying Rules on the Command Line

Rules may be specified on the command line (for example, "--exclude PATTERN") or read from a file (for example, "--exclude-from=FILE") to make async skip or include specific directories and files. The following command-line options can be freely intermixed:

--exclude
--include
--exclude-from
--include-from

Rules are applied in the order that they are encountered, and the first matching rule (whether including or excluding) takes precedence. For example, if the file aspera-test specifies rules W, X and Y, then the following options apply rules V, W, X, Y, Z (in that order).

--include V --exclude-from aspera-test --include Z

Note: Directories and files are visited in strict depth order, so that with the command-line options "--exclude /above/ --include /above/below the file /above/below is never considered. The directory /above/ is not scanned; thus, "/above/below" has no opportunity to be checked against any rules.

Specifying Rules in aspera.conf

Rules may be specified in aspera.conf, as they are for ascp sessions, to apply to sessions run by a specific user or all users. These rules are applied first, then any command line filters are applied.

CAUTION:

Filters set in aspera.conf apply to both ascp and async sessions. If you do not want async filtering rules to apply to ascp sessions, set the rules for a specific user and use that user for async sessions. If you notice your async sessions are being filtered in unexpected ways, search aspera.conf for <filters> to determine what rules have been configured. You can find aspera.conf in the following location:

/opt/aspera/etc/aspera.conf

To specify an inclusion, start the filter pattern with '+ ' (+ and a whitespace, such as + *.jpg). To specify an exclusion, start the filter pattern with '- ' (- and a whitespace, such as - *.png).

To set filters for a specific user, run the following asconfigurator command:

# asconfigurator -x "set_user_data;user_name,username;file_filters,|filter1[|filter2]"

To set filters for all users, run the following:

# asconfigurator -x "set_node_data;file_filters,|filter1[|filter2]"

The separator "|" is not required if only one filter is set.

Individual Matching Rules

Rules applied to directory and file names are limited regular expressions (referred to as "globs" in UNIX terminology). Rules use standard globbing syntax and globbing extensions, as described in the tables below.

Standard Globbing Syntax

Standard Globbing	Description
/	The only recognized path separator.
\	Quotes any character literally, including itself. The `\` character is exclusively a quoting operator, not a path separator.
*	Matches zero or more characters, except a `/` or a`.`preceded immediately by a `/` character.
?	Matches any single character, except a `/` or a `.` preceded immediately by a `/` character.
[ … ]	Matches exactly one of a set of characters, except a `/` or a`.`preceded immediately by a `/` character.
[^… ]	When ^ is the first character, matches exactly one character not in the set.
[!… ]	When ! is the first character, matches exactly one character not in the set.
[`x`-`x`]	Matches exactly one of a range of characters.
`[:xxxxx:]`	For details about this type of wildcard, see any POSIX-standard guide to "globbing".

Examples of Standard Globbing

Standard Globbing	Example	Matches	Does Not Match
/	abc/def/xyz	abc/def/xyz	abc/def
\	abc\?	abc?	abc\? abc/D abcD
*	abc*f	abcdef abc.def	abc/f abc/def abc/.def
?	abc??	abcde	abcdef abc/d abc/.
[ … ]	[abc]def	adef bdef cdef	abcdef ade
[^… ]	[^abc]def	zdef .def 2def	adef bdef cdef /def /.def
[!… ]	[!abc]def	zdef .def 2def	adef bdef cdef /def /.def
[`x`-`x`]	[a-z]def	cdef ydef	Adef 2def .def

Globbing Extensions

Globbing Extensions	Description
/**	Like `*` except that it also matches the `/` character.
* or /** at end of rule	Matches both directories and files.
/ at end of rule	Matches directories only.
no / at end of rule	Matches files only.
/ at start of rule	Matches from the system's root directory (absolute path) only; that is, the entire string must be matched. Note: The `/` means the system's root, not the docroot, and not from the top level specified for the transfer set.

Examples of Globbing Extensions

Globbing Extensions	Example	Matches	Does Not Match
/**	abc/**/def	abc/def abc/x/def abc/.wxy/def abc/wxy/tuv/def	abc/xyz/def/ zabc/wxy/def
* at end of rule	abc*	abc/file abc/dir
/** at end of rule	abc/**	abc/.file abc/dir abc/wxy/.dir abc/wxy/tuv/file	abc/
/ at end of rule	abc/*/	abc/dir	abc/file
no / at end of rule	file	file	dir
/ at start of rule	/abc/def	/abc/def	.../abc/def

Rule Composition

Example	Expected Behavior
--include A --include B	Matches all file or directory names that match A or B.
--include A --exclude B	Matches all file or directory names that match A; of the rest, excludes all that match B.
--exclude B --include A	Excludes all file or directory names that match B; of the rest, matches those that match A.
--exclude-from=FILE1 --include-from=FILE2	Read filter specifications from FILE1 and FILE2 Rules in the file are appended as if from a series of `--include A --exclude B` options Lines that start with `+` or plus, space, (e.g., `+ A`), are equivalent to the option `--include A` Lines that start with `-` or minus, space (e.g., `- B`), are equivalent to the option `--exclude B` Lines that start with `.+` or dot, plus, space (e.g., `.+ F`), are equivalent to the option `--include-from=F` Lines that start with `.-` or dot, minus, space (e.g., `.+ G`), are equivalent to the option `--exclude-from=G` Lines that start with `.` or dot, space, name a file to read for more rules, but without a default (i.e., each line must start with one of the above commands). Other lines default to include (for `--include-from`) or exclude (for `--exclude-from`) rules Leading white space, blank lines and comment lines (`# comment`) are ignored

A file or directory name that does not match any rule is still tracked, as if by a final "+ *" and "+ .*". Also, to reliably exclude all unmatched files, add two final rules: "- *" and "- .*".

Semantics

Excluded new files are invisible to async. Files that have already been synced continue to be tracked even when they have, or get, a name that is now excluded. When run with --exclude FILE3:

Local event	Effect on peer (previously synced)	Clean start
mv FILE4 FILE3	mv FILE4 FILE3	rm FILE4
rm FILE3	rm FILE3	(ignored)
cp FILE4 FILE3	cp FILE4 FILE3	(ignored)
mv FILE3 FILE4	mv FILE3 FILE4	new file FILE4