Using Amazon S3 Versioning with Aspera

The Amazon S3 bucket version feature enables you to retain multiple versions of an object in the same bucket. By default, Aspera uploads overwrite any object that has the same name as a source file. As of version 3.7.3, Aspera transfers can be configured to work more gracefully with S3 versioning.

Prerequisites

  • Versioning must be enabled on the Amazon S3 bucket. For instructions, see:

    http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html

  • HST Server 3.7.3 or later. Before upgrading from a previous version, make a backup and then delete /opt/aspera/etc/trapd/s3.properties. After upgrading, copy your customizations from the backup into the new s3.properties file that was created during the installation process.

Enabling S3 Versioning with Aspera Transfers

The configuration file /opt/aspera/etc/trapd/s3.properties contains the following properties that let Trapd detect if S3 versioning is enabled and change the default behavior of Aspera uploads:

# The period after which a bucket is checked again for cross-region transfers or
# bucket versioning.
# Default is 900000 (15 minutes)
#aspera.session.check-bucket.frequency=900000

# Whether to retrieve the bucket versioning status. To retrieving the bucket
# versioning status one must be the owner of the bucket and have the permission
# "s3:GetBucketVersioning"
# Default is false
#aspera.session.retrieve-bucket-versioning=false

# The assumed bucket versioning status when aspera.session.retrieve-bucket-versioning is false.
# Default is false (assume bucket is not versioned).
#aspera.session.assumed-bucket-versioning=false

To enable S3 versioning:

  • If the user is the owner of the bucket, uncomment aspera.session.retrieve-bucket-versioning and set it to true.
  • If the user is not the owner of the bucket or does not have permission to check the bucket status, set or leave aspera.session.retrieve-bucket-versioning to false. Uncomment aspera.session.assumed-bucket-versioning and set it to true.

Aspera Versioning Behavior

File handling in a versioning-enabled Amazon S3 bucket depends on if the S3 versioning feature in Aspera is enabled.

With Aspera S3 versioning enabled:

  • When a file is first uploaded to the bucket, the object is created with a version ID.
  • When a source file is skipped during an upload because it exists in the bucket, the existing object and its ID do not change.
  • When an object is overwritten in the bucket, the new version of the object is created and assigned a version ID while the original object is left unmodified with the original ID.
    Note: The HST Server GUI does not display all the versions of an object. To view all the versions, use the Amazon S3 console or AWS SDKs.
  • When an object is deleted, the file initially disappears from the HST Server GUI; however, in the Amazon S3 console and AWS SDKs output, the object is replaced with a delete marker. This delete marker becomes the current object version. Previous versions are retained and, upon refresh, the original object reappears in the HST Server GUI.

With Aspera S3 versioning disabled:

  • When a file is first uploaded to the bucket, the object is created with a version ID.
  • When an object is overwritten, the object is replaced by a delete marker and then a new version ID, indicating that the object has been deleted and rewritten.