Aspera Sync Command Reference

Syntax

# async [instance_options] -N pair -d ldir -r [user@host:rdir] [session_options] ...

Required Command Options

Naming the async session: -N pair

Specifying filepaths and filenames: ldir and rdir

ldir specifies the local directory to be synchronized and rdir specifies the remote directory to be synchronized. File paths and filenames must follow these rules:

• The drive letter is required in Windows paths, unless the server's aspera.conf file has a docroot defined for the user. If no drive letter is included when syncing with a Windows machine and docroot is not defined for the user, async displays the error message: "Failed. Peer error: Remote directory is not absolute."
• You can synchronize Windows, Linux, Mac OS X, and other Unix-based endpoints and servers, but must take care with path separators. The path separator "/" is supported on Windows and other platforms. The path separator "\" is platform-agnostic only for the options -d/r/L/R/B/b and --keep-dir-local/remote. In Include/Exclude Filtering Rules, however, "\" is exclusively a quoting operator and "/" is the only path separator recognized.
• File names may not contain \n, \r, or \. Files with these in their names are skipped.
• When scanning or monitoring a file system for changes, async skips over files with names that end in one of the special suffixes specified in aspera.conf with <resume_suffix> and <partial_file_suffix> . To disable this behavior, you can set these values to the empty string. <resume_suffix> defaults to .aspx. The <partial_file_suffix> tag defaults to the empty string, but is often set to .partial.

Specifying the direction of the sync: -K direction

Sync has three modes of synchronization: push, pull, and bidi.

• push: The contents of ldir are synchronized to rdir, with the ldir content overwriting the rdir content, by default (unless the overwrite options are specified otherwise, such as to only overwrite if rdir is older, or never overwrite).
• pull: The contents of rdir are synchronized to ldir, with the rdir content overwriting the ldir content, by default.
• bidi (bi-directional): The contents of ldir and rdir are synchronized, with newer versions of files and directories overwriting older versions in either ldir or rdir, by default.

Using continuous mode: -C

Continuous mode is supported only when the file source is Windows or Linux. See the table below for the operating system requirements for the Sync server and client for the different Sync directions.

One-time synchronization is supported between all operating systems.

Environment Variables

If needed, you can set the following environment variables for use with async. The total size for environment variables depends on your operating system and transfer session. Aspera recommends that environment variable values do not exceed 4096 characters.

ASPERA_SCP_COOKIE=cookie

Set the transfer user cookie. Overridden by --cookie.

ASPERA_SCP_LICENSE=license_string

Set a base64-encoded Aspera license string.

ASPERA_SCP_PASS=password

Set the transfer user password. Overridden by -w and --pass.

ASPERA_SCP_TOKEN=token

Set the transfer user authorization token. Overridden by -W and --token.

Instance Options

-A, --version

Display the async version information and license information.

--apply-local-docroot

Prepend the local docroot to the local directory.

-D[D..]

Log at the specified debug level. Default is 0. Additional Ds return more messages.

-h, --help

Display help for command-line options.

-L log_dir, --alt-logdir=log_dir

Log to the specified logging directory on the local host. If the directory doesn't exist, async creates it for you.

-q, --quiet

Disable all output.

--watchd=datastore:host:port[:domain]

Use asperawatchd connected to the specified Redis for the transfer session. datastore can be redis or scalekv.

For example:

--watchd=redis:localhost:31415

The optional domain argument allows you to specify if the domain is other than the default root. For more information see Working with the Aspera Watch Service.

Session Options

-a policy, --rate-policy=policy

Transfer with the specified rate policy. policy can be fixed, fair, high, or low. Default: fair

--assume-no-mods

Assume that the directory structure has not been modified. If a directory's modification time has not changed compared to the Sync database, async in non-continuous mode skips scanning the directory. This option makes scanning static directory structures faster. Aspera recommends using --exclude-dirs-older-than instead of this option.

-B rdbdir, --remote-db-dir=rdbdir

Save the remote database to the specified directory. Similar to -b, but applies to the remote database. For further usage information, see The Sync Database. Default: .private-asp at the root level of the synchronized directory. The directory is created if it does not already exist. If <async_db_dir> is set in aspera.conf on the server, that setting overrides the location specified with -B .

-b ldbdir, --local-db-dir=ldbdir

Use the specified local database directory. Default: .private-asp at the root level of the synchronized directory.

You can save the Sync database to a different location than the default one under the ldir specified with -d. This allows you to store the database away from the main data files, which is useful for performance tuning. It is also useful when -d ldir is located on a network share volume that does not reliably support database locking. For further usage information, see The Sync Database.

-C, --continuous

Run continuous synchronization. Default: disabled.

Note: Continuous mode is supported only when the file source is Windows or Linux. Continuous pulls can be run from any operating system if the source is Windows or Linux. Continuous push can be run only by Windows or Linux. Continuous bidi requires that both the Sync client and server are Windows or Linux.

If you receive an inotify error when attempting to run continuous synchronization, see Troubleshooting Continuous Mode Errors.

If a file is open, async cannot transfer the file due to sharing violations and may ignore the file if it is closed without changes. To specify the maximum number of retries after a sharing violation, use with --sharing-retry-max. To enable periodic scans that detect when an opened file has been closed and is ready for transfer, use with --scan-interval.

-c cipher, --cipher=cipher

Encrypt file data with encryption algorithm. cipher can be none, AES128, AES192, or AES256. Default: AES128.

--check-sshfp=fingerprint

Compare fingerprint to the remote host key hash and fail on mismatch.

--clean-excluded

Remove excluded directories from snap.db on both Sync endpoints to decrease the size of snap.db. This option applies when directories are excluded by path (--exclude) or by modification time (--exclude-dirs-older-than). If the remote endpoint is running Sync older than 3.8.0, then the option is accepted (the session does not fail) but it has no effect on either endpoint.

--compression={zlib|none}

Compress a file before transfer using the specified method. Default: none.

--cookie=cookie

Specify a user-defined identification string to report to the Aspera Management interface. cookie cannot contain the special characters \r, \n, or \0.

--cooloff=sec

Delay the start of the transfer. For example, if --cooloff=5, async waits 5 seconds before copying a file. If --cooloff=0 transfers start immediately. The client and server use the same cooloff period. Valid range for sec: integers 0-60. Default: 3.

--cooloff-max=sec

Wait up to the specified time (in seconds) for a file to stop changing before skipping synchronization of the file. Using this option prevents a one-time sync from waiting on a constantly changing file. The file is skipped and reported as an error. Default: 0 (disabled).

--create-dir

Create the source directory, target directory, or both if they do not exist, rather than reporting an error and quitting. Use with -d and -r.

-d ldir, --local-dir=ldir

Synchronize the specified local directory. Use --create-dir to create the remote directory if it does not already exist.

--dedup[=mode]

Take the specified the action when Sync detects duplicate files on the source, even if they have different pathnames. Requires -k with a full checksum. Available modes are hardlink, inode, or copy. Default: hardlink.

• hardlink - When two or more source files are duplicates, a hardlink is created between them on the target. This saves storage by preventing multiple copies of the same file from accumulating on the target. The files on the target have the same inode, even if the source files have different inodes. The target must be running a Unix-based operating system. File metadata preservation options (-u and -j) are not supported with this option.
• inode - When two or more source files have matching inodes, a hardlink is created between them on the target and the target files have matching inodes. This is only supported between Unix-based platforms. If --dedup=inode is used in a continuous sync, Aspera recommends using the scan-interval option.
• copy - After a file is synced on the target, the synced file is copied to the duplicate. This saves bandwidth by not transferring duplicate files. This mode is useful when the target is Windows. File metadata preservation options (-u and -j) are supported with this option.

Without the dedup option, all duplicate files are synced. Duplicates might still be synced, rather than hardlinked or copied, if one of the duplicates has not yet been synced on the target.

--delete-delay

Postpone the actual deletion of files or directories until the end of the synchronization.

-E file, --exclude-from=file

Skip paths specified in the filter file. For more information on setting filters, see Include/Exclude Filtering Rules.

--exclude=pattern

Exclude paths that match pattern. Wildcards, such as * and ?, are supported but rules containing them must be in double quotes. For example, --exclude="*.jpg".For more information, see Include/Exclude Filtering Rules.

--exclude-dirs-older-than=mtime

After the initial Sync scan, do not scan directories during subsequent synchronizations if they or their parents have a recursive modified time older than the specified value. The recursive modified time of a directory is the most recent modification time of it or any of its children (file or directory). This option is used to avoid rescanning directories that are known to be unchanged since the previous synchronization, such as a monthly archive directory structure in which only the most recent subdirectory is being modified.

mtime may be specified in any one of the following ways:

• As a positive number of seconds since 1970-01-01 00:00:00, for Unix and POSIX-compliant operating systems.
  Note: Some file servers, such as Windows NT, use a different epoch for the recursive modified time. In this case, MTIME should be specified as a duration relative to present or UTC timestamp.
• As a UTC timestamp with the format YYYY-MM-DDTHH:MM:SS, such as 2015-01-01T08:00:00.
• As a duration formatted as DDd HH:MM:SS or WWw DDd HHh MMm SSs. Directories whose "mtime" is older than Now minus MTIME are not scanned. Input requirements: Leading zero fields and spaces may be omitted. The leftmost fields are optional, but fields to the right of the largest unit specified are required. For example, to exclude directories older than 24 hours, you could specify 1d 0:0:0, 24:00:00, or 24h 00m 00s, but not 1d.

This option does not apply to the root directory.

Note: Sync fails if the first run of async and the next run do not use the same --exclude-dirs-older-than option. If the first run specifies --exclude-dirs-older-than, then the next run must use this option, too. If the first run does not include --exclude-dirs-older-than, then the next run fails if this option is specified.

-G size, --write-block-size=size

Use the specified block size for writing. size=integer K, M, or bytes. Default: 64 MB.

-g size, --read-block-size=size

Set block size for reading. size=integer K, M, or bytes. Default: 64 MB.

-H val, --scan-intensity=val

Scan at the set intensity. val can be vlow, low, medium, high, or vhigh. vlow minimizes system activity. vhigh maximizes system activity by continuously scanning files without rest. Default: medium.

--host=host

Synchronize with the remote host that is specified by hostname or IP address. If the remote host is a cluster, enter the cluster DNS. When using --host=, the characters "@" and ":" are not treated specially in the argument to -r or --remote-dir. The transfer username cannot be specified as part of the remote directory filepath. Instead, it must be set with --user= or in the environment variable $user (on Windows, %USER%). Allowed forms are as follows:

--remote-dir user@host:/rdir # (old method)  
--user user --remote-dir host:/rdir  
--user user --host host --remote-dir /rdir  
--remote-dir host:/rdir  # (uses $user)  
--host host --remote-dir /rdir # (uses $user)

The following means the same as the first three lines above:

 -r /rdir --user=user --host=host

For backward compatibility, -r A:/rdir for any single letter A is still taken as a Windows path, not as --host A -r /rdir. To specify a one-letter host name A, use an explicit --host=A.

-I file, --include-from=file

Scan and include paths specified in the filter file. For more information, see Include/Exclude Filtering Rules.

-i file, --private-key-path=file

Authenticate with the specified SSH private key file. For information on creating a key pair, see Creating SSH Keys.

--ignore-delete

Do not copy removals to the peer. This option is used mostly with uni-directional syncs. In bi-directional sync, a deletion on one side is ignored but the next time async is run, the file is recopied from the other end. In continuous mode, the file is not recopied until either async is restarted or the file is changed (touched).

--ignore-mode

Do not synchronize file permissions of the source to the destination. This argument is useful when synchronizing from a Unix-like source to a Windows destination, which has different file permission behavior than the Unix-like source ("read only" files cannot be deleted or modified on Windows).

--include=pattern

Include paths that match pattern. Wildcards, such as * and ?, are supported but rules containing them must be in double quotes. For example, --include="*.jpg". For more information on how to set include and exclude patterns, see Include/Exclude Filtering Rules.

-j, --preserve-gid

Preserve file owner's gid when synchronizing files between Unix-like operating systems. Requires that async is running as root. Default: disabled.

-K direction, --direction=direction

Transfer in the specified direction. direction can be push, pull, or bidi (bi-directional). Default: push.

-k type, --checksum=type

Calculate the specified checksum type. type can be sha1, md5, sha1-sparse, md5-sparse, or none. A value of none is equivalent to a size check only and async will not detect a change in timestamp. Default: sha1-sparse for local storage, none for object storage.

--keep-dir-local=dir

Move deleted files into dir. The directory must exist (it is not created by --create-dir), and must be outside the synchronization directory (or excluded from the sync using --exclude or --exclude-from), but on the same file system.

--keep-dir-remote=dir

Move the server's deleted files into dir. The directory must exist (it is not created by --create-dir), and must be outside the synchronization directory (or excluded from the sync using --exclude or --exclude-from), but on the same file system.

-l rate, --target-rate=rate

Transfer no faster than the specifed maximum transfer rate. rate=integer G/g, M/m, K/k, or bps. Default: 10 Mbps.

--local-mount-signature=signature file

Verify that the local file system is mounted by the existence of this file. This option increases the time required to synchronize deletes. See also --remote-mount-signature.

-m rate, --min-rate=rate

Attempt to transfer no slower than the specified minimum transfer rate. rate=integer G/g, M/m, K/k, or bps. Default: 200 Kbps.

-N pair, --name=pair

Assign a name for the synchronization session. The value can contain only ASCII alphanumeric, hyphen, and underscore characters. This value is stored in the session cookie and can be used in Aspera Console to identify the transfer session.

Note: -N must precede all session options.

-n action, --symbolic-links=action

Handle symbolic links with the specified method, as allowed by the server. For more information on symbolic link handling, see Symbolic Link Handling.

action can be:

• copy - create or update the link at the destination (default). Not valid for Windows source or destination.
• skip - ignore the link altogether.
• follow - treat the link as if it were the file or directory it points to, so that at the destination, what was a link is now a copy of the file or directory. Functions as skip if source is Windows.

--no-preserve-root-attrs

Disable the preservation of attributes on the Sync root.

--no-scan

Never scan. Use this option in a continuous Sync session to synchronize only new files (files that are added to the directory after the start of the async session) but not existing files. With --no-scan, Sync relies entirely on file system notifications to detect changes. As a result, if a directory is renamed after the Sync session starts, then the directory name is synchronized but the contents are not (because Sync does not recognize that the files were "moved" to the renamed directory). This option cannot be used with --scan-interval or one-time Sync sessions.

-O port, --udp-port=port

Use the specified UDP port for FASP data transfer. Default: 33001.

-o policy, --overwrite=policy

Overwrite files according to the specified policy, which can be always, older, or conflict. Use with -K push and pull. Default: always for -K push and pull; conflict for -K bidi.

Note: When syncing with object storage, only file size (--checksum=none) can be used to compare files. Thus, using --overwrite=always only overwrites files whose sizes have changed. If the content of a local file is different from a file with the same name in object storage but the files are the same size, the file in object storage is not overwritten. To overwrite files in this case, use --overwrite=older.

--overwrite=older is only accurate if the user also specifies --preserve-time (preserve timestamps).

To resolve conflict and error situations in a uni-directional sync, “touch” the problem files on the source and run async with --overwrite=always. This clears all conflict and error states as the problem files are synchronized.

-P port, --tcp-port=port

Use the specified TCP port for SSH. port must be a valid numeric IP port. Default: 22.

--pending-max=N

Allow the maximum number of files that are pending transfer to be no more than the specified number. This option acts as a buffer. Default: 2000.

--preserve-access-time

Preserve file access time from the source to the destination. Default: disabled.

--preserve-acls=mode

Preserve Access Control Lists (ACL) data. mode can be native, metafile, or none. Default: none.

• native - Preserve attributes using the native capabilities of the file system. This mode is only supported for Windows and macOS. If the destination and source do not support the same native ACL format, async reports and error and exits.
• metafile- Preserve file attributes in a separate file, named filename.aspera-meta. For example, attributes for readme.txt are preserved in a second file named readme.txt.aspera-meta. These metafiles are platform independent and can be copied between hosts without loss of information. This mode is supported on all file systems.
• none - Do not preserve attributes. This mode is supported on all file systems.

Important Usage Information:

• This feature is only meaningful if both hosts are in a common security domain. If a SID (security ID) in a source file does not exist at a destination, the synchronization proceeds but no ACL data is saved and the log records that the ACL could not be applied.
• Both --preserve-acls and --remote-preserve-acls must be specified in order for the target side of a pull to apply the ACLs.
• ACLs are not synchronized when only the ACL is modified, or when only the ACL and filename are modified. ACLs are not preserved for directories.
• On Windows, the ACLs that are created for files that are transferred into user directories might restrict file access to specific users. Ensure that the ACLs on the transfer-cache directory (destination_path/session_name) are generic enough to allow access to all users who require it. For more information about the transfer-cache directory, see The Sync Database.

--preserve-creation-time

Preserve file creation time from the source to the destination. Valid only on Windows machines. Default: disabled.

--preserve-modification-time

Preserve file modification time from the source to the destination. Default: disabled.

--preserve-time

Same as -t.

--preserve-xattrs=mode

Preserve file extended attributes data (xattr). mode can be native, metafile, or none. Default: none. If the sync is run by a regular user, only user-level attributes are preserved. If run as superuser, all attributes are preserved. See also --remote-preserve-xattrs.

• native - Preserve attributes using the native capabilities of the file system. This mode is only supported on macOS and Linux. If the destination and source do not support the same native xattr format, async reports and error and exits. If the Linux user is not root, some attributes such as system group might not be preserved.
• metafile- Preserve file attributes in a separate file, named filename.aspera-meta. For example, attributes for readme.txt are preserved in a second file named readme.txt.aspera-meta. These metafiles are platform independent and can be copied between hosts without loss of information. This mode is supported on all file systems.
• none - Do not preserve attributes. This mode is supported on all file systems.

Note: Extended attributes are not synchronized when only the xattr is modified, or when only the xattr and filename are modified. Xattr are not preserved for directories.

--proxy proxy_url

Synchronize using the specified proxy address. The proxy URL is specified with the following syntax:

dnat[s]://proxy_username:proxy_password@proxy_ip_address[:port]

where the default port for DNAT is 9091, and for DNATS is 9092. The Proxy password must be specified or the synchronization fails to connect to the Proxy server.

-R rem_log_dir,--remote-logdir=rem_log_dir

Use the specified logging directory on the remote host. The directory is created if it does not already exist. If <async_log_dir> is set in aspera.conf on the server, async initally logs to rem_log_dir but is then redirected to the directory specified for <async_log_dir>.

Note: -R cannot be used if the transfer user is restricted to aspshell.

-r rdir, --remote-dir=rdir

Synchronize the specified directory on the remote host. rdir is [[user@]host:]path. If the target is the remote directory, you can use--create-dir to create the remote directory if it does not already exist.

CAUTION:

If the source and target directories are both on the local host, do not specify a target directory that is inside your source directory.

--remote-mount-signature=signature file

Verify that the remote file system is mounted by the existence of this file. This option increases the time required to synchronize deletes.

--remote-preserve-acls=mode

Like --preserve-acls but used when ACLs are stored in a different format on the remote computer. Defaults to the value of --preserve-acls.

Note: Both --preserve-acls and --remote-preserve-acls must be specified in order for the target side of the pull to apply the acls.

--remote-preserve-xattrs=mode

Like --preserve-xattrs but used attributes are stored in a different format on the remote computer. Defaults to the value of --preserve-xattrs.

--remote-scan-interval=duration

Set the scanning interval of the remote computer. See also --scan-interval.

--remote-scan-threads=N

Use the specified number of directory scanning threads on the remote computer. More threads decrease the time it takes for async to scan the directory after the initial synchronization, and increase the number of pending files. Default: 1. To specify the number of scanning threads on the local computer, see --scan-threads.

--remove-after-transfer

Remove source files after they are successfully synchronized.

--scan-dir-rename

Enable the detection of renamed directories and files compared to the previous scan, based on matching inodes. Enable the detection of renamed directories and files compared to the previous scan, based on matching inodes. If a new directory is found on the source and its inode matches that of a previously found directory, it is considered a "rename" and the target directory is renamed accordingly. The source directory is scanned for contents change, and the target directory is updated accordingly. This option can only be used on file systems with persistent inodes, and does not work if inodes are volatile, as is the case with many network-mounted file systems.

--scan-file-rename

Enable the detection of renamed files compared to the previous scan, based on matching inodes. If a new file is found and its inode and last-modified time matches that of a previously found file that does not have multiple hardlinks, it is considered a "rename" and the remote file is renamed accordingly. Can only be used on file systems with persistent inodes, and does not work if inodes are volatile, as is the case with many network- mounted file systems.

Note: If --scan-file-rename is used without --scan-dir-rename, a directory rename causes async to create a new directory and rename its files individually.

--scan-interval=duration

Enable periodic scans during a continuous Sync (a session run with the -C option) on the local host. duration is the interval between periodic scans and can be specified as DDd HH:MM:SS.mmm or WWw DDd HHh MMm SSs XXms XXus. Leading zero fields can be omitted. Spaces can be omitted. A plain number XX is interpreted as SSs (seconds).

--scan-threads=N

Use the specified number of directory scanning threads on the local computer. More threads decrease the time it takes for async to scan the directory after the initial synchronization, and increase the number of pending files. Default: 1. To specify the number of scanning threads on the remote computer, see --remote-scan-threads.

--sharing-retry-max=N

Retry synchronizations up to the specifed maximum number after a sharing violation. The interval between retries is the number of seconds specified by --cooloff. Default: 3.

--symbolic-links=action

See -n.

-t, --preserve-time

Preserve file timestamps. Default: disabled.

--tags=string

User-defined metadata tags in JSON format that can be used in transfer session reporting and searches.

--tags64=string

User-defined metadata tags in JSON format and base64-encoded that can be used in transfer session reporting and searches.

--transfer-threads=N[:size]

Use the specified number of dedicated transfer threads and optionally specify the file size at which files are assigned groups of threads. The number of threads should not exceed the number of available CPU cores (the lower value of the client and server machines). If no size is specified, infinity is used as an upper bound.

For example, to use two transfer threads to transfer files smaller than or equal to 128 bytes and six transfer threads for all other files (those larger than 128 bytes), use the following options:

--transfer-threads=2:128 --transfer-threads=6

-u, --preserve-uid

Preserve the file owner's uid when synchronizing files between Unix-like operating systems. async must be run as root to use this option. Default: disabled.

--user=user

Authenticate the transfer with the specified username. With this option, the characters "@" and ":" are not treated specially in the argument to -r or --remote-dir.

-W token_string, --token=token_string

Use the specified authorization token. The token type (sync-push, sync-pull, or sync-bidi) must match the direction (push, pull, or bidi) of the requested transfer. The token path must match the remote directory of the requested transfer. If an invalid token is provided, the requested transfer will be denied.

-w pass, --pass=pass

Authenticate the transfer with the specified password.

--write-uid=uid

--write-gid=gid

Write files as the user uid or the group gid. uid and gid can be numeric, or by name. If by name, the name is looked up on the hostperforming the write. Failure to set the uid or gid is logged, but is not an error. The uid or gid is set after ascp completes and before moving the file from the staging directory to the final location.

--write-uid conflicts with --preserve-uid, and --write-gid conflicts with --preserve-gid.

-X size, --rexmsg-size=size

Use the specified size (in bytes) for a retransmission request. Maximum: 1440.

-x, --reset

Clear the async database and rescan the synchronized directories and files to create a fresh database. Default: off.

-Z mtu, --datagram-size=mtu

Use the specified datagram size. Value is an integer. Default: detected-path MTU.