Customizing Cluster Nodes Using First-boot Scripts

To modify the default cluster node images provided by Aspera, you can enter a first-boot script into the Firstboot Script field when launching a node.

The script is executed during the initialization process of each cluster node, before the default Aspera first-boot sequence and before the Cluster Manager starts (but not when a node instance restarts). Because no Aspera services are running at this time, any configuration changes (such as modifications to trapd.properties) can be executed without needing to restart the services.

Note: If you are making modifications that involve lengthy software installations, modify the Cluster Manager and cluster node images instead. For more information, see Customizing the Cluster Manager or Cluster Node Images.

The following example scripts configure users and the system, and can be combined or expanded to include other configuration settings.

Print a Test Message

Use the following script to print a test message to confirm that the first-boot scripts are running:

#!/bin/bash

echo "Cluster Manager first-boot scripts are running."

Add a User to the Cluster Manager

Use the following script to add a new user and configure their SSH key. Replace public_key with the text string in /opt/aspera/var/aspera_tokenauth_id_rsa.pub (the server's public SSH key).

Note: The new user must be manually added to the sshd configuration file in the Cluster Manager web UI; this cannot be done as part of the first-boot script. Instructions follow the script example.

#!/bin/bash

useradd username
mkdir -p /home/username/.ssh
echo "public_key" > /home/username/.ssh/authorized_keys
chown -R username:username /home/username/.ssh/
chmod 700 /home/username/.ssh/
chmod 600 /home/username/.ssh/authorized_keys

Adding the user to the sshd configuration: Once the cluster is updated with the new user, log in to the Cluster Manager web UI as an admin, go to the Clusters page, and click Action > Edit Cluster Configuration for your cluster. Locate the following text:

"sshd_transfer_config":...AllowUsers xfer xfer1 xfer2 xfer3\n\n...

Add the new username to the list of allowed users. For example:

AllowUsers xfer xfer1 xfer2 xfer3 username\n\n

Click Save Changes.

Create a Node API User and Password

Use the following script to create a Node API username (xfer2) and password that can be used to monitor your clusters by using IBM Aspera Console. Replace node_api_password with your desired password.

#!/bin/bash

echo "/opt/aspera/bin/asnodeadmin -a -u xfer2 -x root -p node_api_password --acl-set impersonation" >> /opt/aspera/atcm/bin/configure-node-users.sh

Note: For more information about the Node API User in Console, see the IBM Aspera Console Admin Guide.

Create a New Default System User to Use with Access Keys

The cluster connects to cloud storage using access keys, which provide a more secure and flexible alternative to authenticating with the Aspera node user or system user. For more information about access keys, see Access Key Overview.

Aspera recommends creating a new system user to use with all access keys. To create this system user, use the first-boot script and replace username with the desired name of your default user:

#!/bin/bash

SYSTEM_USER=username

function createUser() {
  local user=$1
  groupadd -f aspusers
  adduser -s /bin/aspshell -G aspusers ${user}
  pushd /home/${user}
    mkdir -p .ssh
    cp -f /opt/aspera/var/aspera_id_dsa.pub .ssh/authorized_keys
    chown -R ${user}:${user} .ssh/
    chmod 755 .ssh/
    chmod 644 .ssh/authorized_keys
  popd
}

createUser "$SYSTEM_USER"

echo "
/opt/aspera/bin/asnodeadmin -d -u \"\$3\"
/opt/aspera/bin/asnodeadmin -a -u \"\$3\" -x \"$SYSTEM_USER\" -p \"\$4\"" >> /opt/aspera/atcm/bin/configure-node-users.sh

Enable HTTPS

Use the following script to automatically use HTTPS for connections to your object storage:

#!/bin/bash
        
sed -i 's/#s3service.https-only=false/s3service.https-only=true/' /opt/aspera/etc/trapd/s3.properties

Enable Cluster Manager to Use S3 Versioning

Use the following script to enable S3 versioning in the Cluster Manager:

#!/bin/bash

sed -i 's/#aspera.session.assumed-bucket-versioning=false/aspera.session.assumed-bucket-versioning=true/' /opt/aspera/etc/trapd/s3.properties

Enable Object Symlinking

Use the following script to enable the Aspera Object Symlink feature in the system-wide configuration.

#!/bin/bash
        
sed -i 's/#aspera.session.support.symlink = false/aspera.session.support.symlink = true/' /opt/aspera/etc/trapd/s3.properties

Disable the Region Check from your S3 Bucket

Use the following script to disable the region check for your S3 bucket. Though discouraged, use it if you are downloading data from public S3 buckets, in which case you are not permitted to verify the region of the bucket through the API.

#!/bin/bash

sed -i 's/#aspera.session.check-bucket.transfer=true/aspera.session.check-bucket.transfer=false/' /opt/aspera/etc/trapd/s3.properties

Disable Content Type Assignments to S3 Objects

Aspera automatically applies Content Type to S3 objects that are uploaded through an ATC node, according to the list in /opt/aspera/etc/trapd/mime-types.props. Use the following script to disable content type assignments.

#!/bin/bash
        
sed -i 's/#aspera.mime-types.enabled=true/aspera.mime-types.enabled=false/' /opt/aspera/etc/trapd/trap.properties

Automatically Tag New Elastic Block Store (EBS) Volumes

Use the following script to automatically tag EBS volumes created by cluster nodes. Replace the values for tag_name and tag_value.

#!/bin/bash
        
Key=tag_name
Value=tag_value
Region=$(curl -sS http://169.254.169.254/2014-11-05/dynamic/instance-identity/document/ | jq --raw-output '.region')
Instance=$(curl -sS http://169.254.169.254/latest/meta-data/instance-id)
Volume=$(aws ec2 describe-instances --instance-ids $Instance --region $Region | jq --raw-output '.Reservations[].Instances[].BlockDeviceMappings[].Ebs.VolumeId')

aws ec2 create-tags --resources $Volume --tags Key=$Key,Value=$Value --region $Region