Sync to S3 on AWS Cloud Object Storage

Sync can be used to sync files when the source or destination is S3 on AWS Cloud Object Storage.

Capabilities:

Requirements:

  1. SSH into your instance as root by running the following command.
    The command is for Linux but also works for Mac. Windows users must use an SSH tool, such as Putty.
    # ssh -i identity_file -p 33001 ec2-user@ec2_host_ip
  2. Elevate to root privileges by running the following command:
    # sudo su -
  3. Set an S3 docroot for the system account user that will be used to run Sync.
    # asconfigurator -x "set_user_data;user_name,username;absolute,s3://s3.amazonaws.com/bucketname"

    If you are not using IAM roles, then you must also specify the S3 credentials in your docroot:

    s3://access_id:secret_key@s3.amazonaws.com/my_bucket
  4. Set database and log directories for async by running the following commands:
    # asconfigurator -x "set_node_data;async_db_dir,/mnt/ephemeral/data/db"
    # asconfigurator -x "set_node_data;async_log_dir,/mnt/ephemeral/data/log"

    This specifies that the logs and Sync database are stored in no-cost ephemeral storage associated with your instance.

Sync Examples:
Note: If the client is on the cloud storage host, the following options are required:
  • The log directory and local database directory should be specified using the following options:
    -L /mnt/ephemeral/data/log -b /mnt/ephemeral/data/db
  • For PUSH and BIDI modes, async must use the apply-local-docroot option.

A one-time (non-continuous) sync with a local disk pushing to an S3 bucket using SSH keys (for more information on using SSH keys, see Creating SSH Keys):

# async -N sync-to-s3 -d /data/data-2017-01 -r bobcat@192.0.4.24:/data -i /bobcat/.ssh/private_key -K push -B /mnt/ephemeral/data/db

A one-time bidirectional sync run from the S3 client to a local disk:

# async -L /mnt/ephemeral/data/log --apply-local-docroot -N bidi_london -d /data -r bear@192.0.12.442:/data -K bidi -b /mnt/ephemeral/data/db -B /async/log