Sync with AWS S3

Sync can be used to sync files when the source or destination is AWS S3 Cloud Object Storage.

Capabilities:

Requirements:

  1. SSH into your instance as root by running the following command.
    The command is for Linux but also works for Mac. Windows users must use an SSH tool, such as Putty.
    # ssh -i identity_file -p 33001 ec2-user@ec2_host_ip
  2. Elevate to root privileges by running the following command:
    # sudo su -
  3. Set an S3 docroot for the system account user that will be used to run Sync.
    # asconfigurator -x "set_user_data;user_name,username;absolute,s3://s3.amazonaws.com/bucketname"

    If you are not using IAM roles, then you must also specify the S3 credentials in your docroot:

    s3://access_id:secret_key@s3.amazonaws.com/my_bucket

    By setting the docroot for the system user, the account becomes an Aspera transfer user. For more information on configuring transfer users, see the Enterprise Server Admin Guide for Linux: Managing Users from the Command Line.

  4. Set database and log directories for async.
    These directories must be located in /mnt/ephemeral/data. The /mnt/ephemeral/ directory is no-cost ephemeral storage that is associated with your instance. Aspera recommends creating a directory to use that is named for the transfer user, and giving the transfer user write access. For example, if the transfer user is ec2_user, run the following commands to create the directory /mnt/ephemeral/data/ec2_user, create the database and log subdirectories, give ec2_user write access, and set the directories as the location for the database and logs:
    # mkdir /mnt/ephemeral/data/ec2_user
    # mkdir /mnt/ephemeral/data/ec2_user/db
    # mkdir /mnt/ephemeral/data/ec2_user/log
    # chown -R ec2_user /mnt/ephemeral/data/ec2_user
    # asconfigurator -x "set_node_data;async_db_dir,/mnt/ephemeral/data/ec2_user/db"
    # asconfigurator -x "set_node_data;async_log_dir,/mnt/ephemeral/data/ec2_user/log"